Open source DevOps Tools

DevOps Tools

1. Nagios (& Icinga)

Infrastructure monitoring is a field that has so many solutions… from Zabbix to Nagios to dozens of other open-source tools. Despite the fact that there are now much newer kids on the block, Nagios is a veteran monitoring solution that is highly effective because of the large community of contributors who create plugins for the tool. Nagios does not include all the abilities that we had wanted around the automatic discovery of new instances and services, so we had to work around these issues with the community’s plugins. Fortunately, it wasn’t too hard, and Nagios works great.

We also looked into Icinga, which was originally created as a fork of Nagios. Its creators aim to take Nagios to the next level with new features and a modern user experience. There is a debate within the open source community about the merits of Nagios and its stepchild, but for now we are continuing to use Nagios and are satisfied with its scale and performance. The switch to newer technology, such as Icinga, may be appropriate in the future as we progress.


2. Monit

Sometimes the simplest tools are the most useful, as proven by the simple watchdog Monit. Its role is to ensure that any given process on a machine is up and running appropriately. For example, a failure occurs in Apache, Monit will help to restart the Apache process. It is very easy to setup and configure and is especially useful for multi-service architecture with hundreds of micro-services. If you are using Monit, make sure to monitor the restarts that it executes in order to surface problems and implement solutions (rather than just restarting and ignoring the failure). You can do this by monitoring Monit’s log files and ensuring that you are alerted to every restart.


3. ELK – Elasticsearch, Logstash, Kibana – via Logz.io

Stack is the most common log analytics solution in the modern IT world. It collects logs from all services, applications, networks, tools, servers, and more in an environment into a single, centralized location for processing and analysis. We use it for analytical purposes (e.g., to troubleshoot problems, monitor services, and reduce the time it takes to solve operational issues). Another use for this tool is for security and auditing (e.g., to monitor changes in security groups and changes in permissions). After receiving alerts on these issues, it is easy to act on unauthorized users and activities. We also use ELK for business intelligence, such as monitoring our users and their behavior. You can set up your own ELK or buy it as-a-service. We’ve written a guide for the community on using ELK to monitor your application performance.



4. Consul.io

Consul is a great fit for service discovery and configuration in modern, elastic applications that are built from microservices. The open-source tool makes use of the latest technology in providing internal DNS names for services. It acts as a kind of broker to help you sign and register names, enabling you to access service names instead of specific machines. If, for example, you have a cluster of multiple machines, you can simply register them as a single entity under Consul and access the cluster easily. We praise this tool for its efficiency, although we still feel there is more that can be done with it. If you also use it, it would be great to hear about your own use case.


5. Jenkins

Everyone knows Jenkins, right? It’s not the fastest or the fanciest, but it’s really easy to start to use and it has a great ecosystem of plugins and add-ons. It is also optimized for easy customization. We have configured Jenkins to build code, create Docker containers (see the next item), run tons of tests, and push to staging/production. It’s a great tool, but there are some issues regarding scaling and performance (which isn’t so unusual). We’ve explored other cool solutions such as Travis and CircleCI, which are both hosted solutions that don’t require any maintenance on our side. For now, however, since we’ve already invested in Jenkins, we’ll continue with it.


6. Docker

Everything that can be said about how Docker is transforming IT environments has already been said. It’s great…life changing, even — (although we’re still experiencing some challenges with it). We use Docker in production for most services. It eases configuration management, control issues, and scaling by allowing containers to be moved from one place to another.

We have developed our SaaS solution with a twelve-layer pipeline of data processing. Together with Jenkins and Docker, we have been able to run a full pipeline across all layers on a single Mac. It would be wrong to say that there aren’t any complications with Docker, as even small containers can take a significant amount of time to build. However, we want to ensure that our developers are as satisfied as possible and enable them to work rapidly. With all of the management involved in storage, security, networking — and everything surrounding containers — this can be a challenge.

We see Docker progressing and look forward to welcoming the company’s new management and orchestration solutions. For those who might be having issues with Docker, we’ve also compiled a list of challenges and solutions when migrating to Docker.


7. Ansible

Again, simplicity is key. Ansible is a configuration management tool that is similar to Puppet and Chef. Personally, we found those two to have more overhead and complexity to our use case– so we decided to go with Ansible instead. We know that Puppet and Chef probably have a richer feature set, but simplicity was our desired KPI here. We see some tradeoffs between configuration management using Ansible and the option to simply kill and spin new application instances using a Docker container. With Docker, we almost never upgrade machines but opt to spin new machines instead, which reduces the need to upgrade our EC2 cloud instances. Ansible is used mostly for deployment configuration mostly. We use it to push changes and re-configure newly-deployed machines. In addition, its ecosystem is great, with an easy option to write custom applications.



8. Collectd/Collectl

Collectd/l are nifty little tools that gather and store statistics about the system on which they run and are much more flexible than other tools. They allow users to measure the values of multiple system metrics and unlike other log collection tools that are designed to measure specific system parameters, Collectd/l can monitor different parameters in parallel. We use these two tools to measure customer performance parameters and ship them to our ELK-as-a-Service platform. We’ve specifically wrapped a Collectl agent in a Docker container and push it with Ansible to all of our servers. It collects information every couple of seconds and then ships it to ELK to allow us to run reports and send alerts. If you’d like to see a specific example of how we do this process in our environment and how others can do the same, we’ve created a guide for everyone.



9. Git (GitHub)

githubGit was created 10 years ago following the Linux community’s need for SCM (Source Control Management) software that could support distributed systems. Git is probably the most common source management tool available today. After running Git internally for a short period of time, we realized that we were better suited with GitHub. In addition to its great forking and pull request features, GitHub also has plugins that can connect with Jenkins to facilitate integration and deployment. I assume that mentioning Git to modern IT teams is not breaking news, but I decided to add to it to the list due to its great value to us.

DevOps

A clipped compound of development and operations is a culture, movement or practice that emphasizes the collaboration and communication of both software developers and other information-technology (IT) professionals while automating the process of software delivery and infrastructure changes.

Correlation in JMeter

Correlation in JMeter

What is correlation and why it is required?
Correlation is the most important aspect of scripting. It generally includes fetching dynamic data from preceding requests/calls and posting it to the subsequent requests.

Let's take an example to find out why exactly we need correlation-
Suppose we have recorded a scenario in which -
>User enters login details and click OK button
>Home page opens and user take further actions

Now, if we just playback this script, the test will fail even for a single user. This is because of the authentication mechanism used. When we login to a website, session variables are dynamically created. These session variables are passed to the subsequent requests and help validation and authentication of the actions performed. So, one cannot just record and playback the requests having these variables. Here, we need to correlate the web requests with the dynamic variables. And for correlation, we need to use the "Regular Expression Extractor" which makes use of regular expressions.

A brief insight to regular expressions-
Regular expressions are used to fetch data from a string based on a search pattern. Basically, what we do is- in order to extract any value (generally a dynamically created value) from a string (text response), we define a left bound of the variable then some wildcard characters and then right bound- (Left Bound)(Wildcard Characters)(Right Bound)

E.g. for if we have text response like-
.......__EVENTVALIDATION"value="weriudflsdfspdfusdjfsisdpfjpsdfohsdihffgdfgpdfjsdjfpj" />...
And we need to extract the value of Event validation variable using regular expressions; the regular expression for the same will be-
__EVENTVALIDATION" value="(.+?)" />
where, Left Bound = __EVENTVALIDATION" value="
Wildcard characters = (.+?)
Right Bound = " />

If you do not want to get deeper into regular expressions, then the wildcard characters (.+?) would suffice in most of the cases. For more information on regular expressions and meaning of each wild card character visit http://www.regular-expressions.info/tutorialcnt.html.

Regular Expression Extractor-

Coming back to JMeter, consider an example where we have two operations-
1. User launch website
2. User fill details and click on OK button
Now, the call user launch website creates a dynamic variable event validation that we can check in Response Data tab of "View Result Tree" listener for the call. The value of this variable is then passed to subsequent call related to "User fill details and click on OK button" as Http post parameter.

Steps for correlating the Event validation values-
1. Run the script containing the both the above stated operations
2. Go to Response tab (Text mode) in "View Result Tree" listener of "User launch website" operation. BTW, we see the second operation "User fill details and click on OK button" in red because it is not yet correlated.

See more at: http://testingattheedge.blogspot.in

3. Create a Regular expression for extracting the value of Event validation variable's value. As stated above the Regular Expression. for this will be- __EVENTVALIDATION" value="(.+?)" />

4. Go to http request under "User Launch Website" transaction controller-> Add -> Post Processor -> Regular Expression Extractor.

Adding "Regular Expression Extractor" control

Regular Expression Extractor Parameters Filled

5. The reference name inserted is the name of the variable created that will capture the Event validation value generated by the http request under "User launch website" operation.

6. Now pass this variable to the subsequent http request under "User fill details and click on OK button" as post request- overriding the already present hard-coded value of Event Validation variable.

Request without correlation (Hard-coded values)

Request with correlation (Dynamic values)

7. Run the Test plan again. All green? That's it.

Customers report strange hissing sound when the iPhone 7 is under stress

THOSE lucky enough to get their hands on a new iPhone 7 have been reporting a strange defect with the device.


When the handset is pushed to its processing limits it has a hissy fit, according to users. Basically, if the phone is working overtime by, for example, running lots of applications at once, it begins to make a faint hissing sound.

The noise was first pointed out by Stephen Hackett at 512 Pixels and was quickly followed by other users reporting the same thing.

In terms of malfunctions it absolutely pales in comparison to the exploding battery issue in Samsung’s Note 7 that prompted a global recall recently. But nevertheless, Apple customers and technologists have been debating the exact cause of the curious hissing sound.

“Some suspect coil whine or similar electromagnetic effects, but there’s no guarantee that this is the case,” claimed Jon Fingas from engadget.

Not all news phones are making the unsettling noise though. A number of customers reported putting their iPhone 7 under enormous stress and heard no hissing sound at all, leading some to speculate the problem could be a manufacturing issue rather than an inherent design quirk.

Samsung begins to replace Galaxy Note 7

Samsung Electronics has begun delivering the new Galaxy Note 7 to users after a worldwide recall following reports of several handsets exploding.

"The exchange program began today (Monday) and is being carried out without problems," a Samsung spokesperson told EFE.

Note 7 users who had returned their old devices will receive a new smartphone of the same model and color unless they opted for a refund.

In South Korea, the exchange is taking place 17 days after the company announced the recall and will take place in other parts of the world in the coming days, except Canada and Singapore, where the company began handing out the new devices last week.

The new Galaxy Note 7 has a green battery indicator that separates it from the earlier version.

Of the 2.5 million Note 7 sold worldwide since its launch on August 19, around 400,000 were sold in South Korea and one million in the US.

Samsung recalled Note 7 on September 2 after admitting that in 35 cases the devices had caught fire while they were being charged owing to faulty batteries.