DevOps is not a thing! — Part II
The working aspect of a DevOps engineer is vast hence this article has been divided in two parts so as to maintain the readability. This is the Part II of the article. If you wish to read the first half of this article then follow the below link.
CI/CD
CI stands for Continuous Integration and CD stands for Continuous Deployment.
With the introduction of CI/CD we are entering into the world of automation. Which is very much essential to make a DevOps person’s life a bit relaxing or else they would be sitting day-night in front of their machines.
In software development there is a need for very frequent code changes especially in today’s world where there is cut-throat competition. Now to deploy the piece of code there may be certain steps which needs to be followed every time. It may include creating a new build, running unit test cases, putting it to a build artifact repository, then deploying it to the VM, then running an end-to-end test. These tasks are quite repetitive in nature and take a considerable amount of time of a person also it is prone to human errors if done manually. But all of this can be automated with the help of CI/CD pipeline.
Many times these terms would be used interchangeably because there is a very subtle difference between them. CI makes sure that the new set of code that is going to be merged won’t affect the existing set of code; such as making sure of certain code practices/methodologies, running a set of unit test cases, testing for the creation of a successful build. Whereas CD happens after CI, it ensures that that new set of code that has been integrated gets deployed on the designated software environment according to the defined CD instructions.
Tools of trade:
- AWS CodePipeline
- Bitbucket pipeline
- Circle CI
- Tarvis CI
- Bamboo
- Jenkins
- Azure devops pipeline
Resource to extend your knowledge:
- Gitlab CI pipeline tutorial for beginners
- What is Jenkins | Jenkins Tutorial for Beginners | Jenkins Continuous Integration Tutorial | Edureka
Tip: Most of the concepts of modern CI/CD pipelines are the same. If you learn one, then there is not much of a difference in another one. So learn the concept of setting up a pipeline rather than learning one specific platform. Because at the end of the day it would be a YAML configuration.
Monitoring
Monitoring is as crucial as setting up the above services. Suppose you have set up an EC2 instance but it has started to cause issues but you are not able to tell what is causing the issue. You need some or the other way to tell that there is an underlying issue and point out what it is. It may be the IOPS consumption, or low bandwidth, or high CPU utilization. With a proper set of monitoring tools one can proactively trace the underlying issues and remediate them as fast as possible.
You may think I can manually monitor these, but think when you have a cluster of 100 to 1000 instances, so it makes a lot of sense to create a control station to monitor everything.
Tools of trade:
- ELK stack (Elasticsearch, Logstash, Kibana)
- Prometheus
- Grafana
- Datadog
- Nagios
- Cloudwatch
- Sentry
- Rollbar
Resource to extend your knowledge:
- Amazon CloudWatch Tutorial | AWS Certification | Cloud Monitoring Tools | AWS Tutorial | Edureka
- Setting up Prometheus and Grafana for monitoring your servers
- Sentry
- Rollbar
Tip: Setting this early in your setup relieves a lot of pain by the time your architecture matures. This might be overlooked by your organization at first but eventually a time will come when they reach a common consensus.
Infrastructure-as-code or IaC
So in terms of a DevOps’ job it is a hell lot of work as we have seen so far and there is always a room for error whenever things are being done manually. After all humans are prone to error there is no denying that statement. As you will be working a lot around your infrastructure and you might be constantly making changes to it and there is always a chance of error while making any changes to it. What if your one incorrect configuration causes hours or outage to your organization’s process or products or even worse a data leak. So in order to reduce these issues, a new concept has emerged in the market where all your infrastructure needs are maintained in a Git like repository pattern. These IaC services provide an interface to your cloud platform. You store your configuration in manifests like files and apply those changes, all you need to make sure is that the manifest is properly created. So instead of pushing a lot of buttons all you need to do is to run a script. The other benefit of this is that you can always replicate your infrastructure with a few clicks instead of doing everything manually. This is extremely helpful in situations where you need to scale your infra without any manual intervention.
Tools of trade:
- CloudFormation
- Terraform
- Chef
- Puppet
- Ansible
Resource to extend your knowledge:
- Terraform tutorials
- Get started with AWS CloudFormation
- Ansible for beginners
Securing your resources
If you have spent a considerable amount of time in the software industry you must have encountered news where an organization got hacked with Ransomware, data leaks, DDoS attacks.
If you take the recent case of Garmin attacks, they had to pay a massive $10 million to the hackers to be back online with their processes. Garmin is one of the top tech giants in the world and I believe they have a considerable amount of budget for such security measures despite that they got hacked. So we need to learn one thing for sure is that there is always some kind of vulnerability lurking in our infrastructure or code and it is just a matter of fact for it to be discovered before being exploited. Therefore it should ring all the bells when taking security at top of your concerns. Because if not invested early on then these things can lead to massive losses.
“Enough of scaring us, what should we do?”
There are a couple of things that you can do
- Following production checklist — This is extremely important (feels kinda obvious but gets overlooked). This is coming from my personal experience. Once I hosted a testing MongoDB server and while I had created the users in the database but I did not follow that by default MongoDB is open for connections without enabling authorizations. The next morning when I woke up my DB was hacked by a ransomware, luckily it was a test DB so I could just chuck it out as no important info was there. But then I discovered it’s production checklist where they clearly mentioned this. So it is extremely important that you follow whatever tech you use, you follow the production checklist or guidelines.
- Following best practices — most of the time when there is a hack it is mostly related to the configurations that have been implemented. We have all seen that a lot of times S3 buckets have been leaked because of poor configurations so make sure that you follow best practices governed by the cloud provider
- OWASP Vulnerabilities — Lookout for top 10 OWASP vulnerabilities they are the most common of the attacks or vulnerabilities found. Follow the cheatsheet and you should be fine. But remember this is not only related to infrastructure but the codebase too. So these points should be kept in mind while developing the solution
- Get hacked — Getting audited by a team of ethical hackers I can assure you that there are more ways to get hacked than you can think of because these hackers think backwards they make things go up-side-down sometimes in a matter of seconds. So get your platform tested if the budget allows.
A lot of these things come from another role called DevSecOps but following these rules should be more than enough to get you covered and the rest of it can be upgraded as you evolve with the help of firewalls and more restricted access to your developers etc.
Tools of trade:
Resource to extend your knowledge:
- Detectify
- OWASP top 10
Software development life cycle or SDLC
After all you are in the software development business you need to have this knowledge because a lot of your work can be divided into various phases of the SDLC. For example in the above point we discussed that a lot of vulnerabilities are on the code side alone. So with your knowledge of the SDLC you may govern the rules in the design or tech discussion phase of the module which can give greater control and security of it in the long run.
There has been a new practice in the town called Shift-Left where software defects and issues can be identified early before the delivery of the software. So unless you know how the whole development process works you can’t just go ahead help your team.
Resource to extend your knowledge:
The working aspect of a DevOps engineer is vast hence this article has been divided in two parts so as to maintain the readability. This was the Part II of the article. If you wish to read the first half of this article then follow the below link.
This article was created in efforts to introduce the newbies in the field of DevOps as in various channels and forums I have encountered various confusions that they have and don’t have a very picture of it. If you wish to improve this article I welcome your suggestions. Some of the topics have just been touched on a very surface level so if you do not agree with its facts then please let me know I will correct it :)
If you have more to discuss then please comment below or we can discuss this over my Instagram handle: i_m_vibh or my Twitter handle: VibhanshuBiswas
