Joe Miller
Ops/Engineering. Continuous DevOping at Webscale
Twitter
Github
The first and obvious option for shipping logs from a heroku app to Logstash is the heroku input plugin. However, this requires installing the Heroku gem and deploying the login + password of a Heroku user to your Logstash server(s). At this time it seems that any user given permissions to an app on Heroku has full control. Not good when you just want to fetch logs. Heroku has added more granular permissions via OAuth but the Heroku gem does not support OAuth tokens yet.
Fortunately there’s another option using Heroku’s log drain. Setting up a log drain from Heroku to Logstash is almost as simple as the Heroku input plugin but has the major advantage of not requiring any new users or passwords to be deployed on the Logstash server.
In a previous post I described two methods for routing metrics generated by Sensu clients to Graphite:
- use a
pipe handler to send metrics via TCP to graphite
- use Graphite’s AMQP (rabbitmq) support.
Method #1 was simply described for completeness. It is not scalable and shouldn’t be used except for very small workloads. Pipe handlers involve a fork() by sensu-server for every metric received.
At the time I recommended method #2 which was more efficient - Sensu would simply copy the metric from its own results queue to another queue that Graphite would be listening on, since both Sensu and Graphite can talk to RabbitMQ.
Here’s how to get Dell OpenManage running on Fedora 19. It is a simple process but I could not find any up to date information on how to do it elsewhere.
I was invited to speak at CentOS Dojo in Phoenix, AZ recently (May 2013) about the Sensu monitoring framework. I wanted to do something a little bit different than past presentations and try to show some use cases that fit what Sensu can do rather than just do a basic introduction to Sensu.
Simple hack for speeding up vagrant provisions.
In past articles we have covered some of basics of Sensu handlers. A nice feature we haven’t touched on yet is handler “sets”. Handler sets were added around v0.9.2 and can be quite useful for saving time when modifying your handler.
If you have read @ripienaar’s excellent series of articles on common messaging patterns you probably noticed a handy CLI tool for working with STOMP queues called stompcat. I looked around for something similar for AMQP brokers but couldn’t find anything quite the same. There is amqp-utils but I had some issues with these and the tools didn’t work quite like I was hoping. So I wrote amqpcat with the idea of providing a similar tool to stompcat.
Available on github and rubygems.org: https://github.com/joemiller/amqpcat
Updated December 7, 2013 : I no longer recommend using the approach described in this post. Please read Sensu and Graphite, Part 2 instead.
In my previous article I mentioned a key strength of Sensu is the ability to re-use existing Nagios plugins. This is a powerful feature of Sensu. Nagios has been around for at least 1000 years according to most recent archaeological discoveries, which means a vast amount of human effort (and capital) has gone into creating Nagios plugins. Being able to leverage this prior effort is a huge win. In this article I’ll demonstrate creating a Sensu check with the check_http Nagios plugin.
(5/15/2012) NOTE: This guide has been superseded by the official ‘ Install Guide’ doc on the Sensu wiki. The new process utilizes the simpler Omnibus-style Sensu packages and covers installation on Debian/Ubuntu platforms as well. Please use this guide instead of the instructions below.
Sometimes it is pretty obvious when Puppet changes something in your infrastructure and bad things happen in a big dramatic way. Other times it’s not so obvious. It can be invaluable to be able to correlate changes made by Puppet to other events happening in your infrastructure.
Statsd is a simple client/server mechanism from the folks at Etsy that allows operations and development teams to easily feed a variety of metrics into a Graphite system. For more info on statsd read the seminal blog article on Statsd “Measure Anything, Measure Everything”.
Previously, I wrote a post about using the ‘dummynet’ functionality in Mac OSX’s ipfw(8) firewall to simulate a variety of networking conditions, such as: bandwidth, packet loss, latency (delay). This is a great feature for testing software under a variety of network conditions but it can be a little tough to use unless you’re comfortable at the command line, or even better, have unix scripting skills since there are multiple commands required to create even simple scenarios.
Then, today I noticed that Apple now includes a new prefPane in Xcode 4.1 and Lion called “Network Link Conditioner” that simplifies all of this, and even includes a few profiles to get you started (eg: “Wifi, Average case”, “3G, Lossy Network”.) Pretty cool feature. Especially useful for iOS developers. Screenshot below.
In an attempt to learn some Groovy and Gradle I wrote an implementation of a Statsd client in groovy. It’s similar to other statsd clients in other languages and supports the typical increment(), decrement(), and timing() methods.
On May 26, 2011, John Vincent, aka @lusis, started a conversation with a simple tweet that led to an hour-plus chat on IRC freenode between some very smart folks. The IRC channel was supposed to go away after the chat was over but instead it has survived and is attracting more and more attention.
The consensus at this point is that Skype 5 on Mac OSX is ugly and wastes a ton of screen space for no good reason (all of this is true, btw.) But, you don’t have to live with the ugliness. It’s not widely known, but Skype 5 supports skins or “chatstyles”, which are constructed entirely in HTML/CSS/Javascript, so they’re easy to write and hack.
I built a Charlie Sheen “Persona” plugin for our Hudson continuous integration server at work, which is now available on my github page. If you’re familiar with the Chuck Norris plugin, then you know what this is all about.
Add a little bit of Sheen-ius to your Hudson/Jenkins server!! Tiger blood is included. #winning
I am a fan of using Kanban boards to organize and visualize the work coming into an infrastructure team, but most of the Kanban resources on the internet are geared towards software development teams.
Recently, a new mailing list was started for the sole purpose of sharing Kanban experiences in operations teams. The list is appropriately named kanbanops and you can sign up, browse the archives, etc, over at yahoo groups:
Collectd is a powerful tool for gathering metrics using its wide range of plugins, such as cpu, disk, load, memory, etc. But there is a lack of good frontend tools for visualizing the data collectd produces.
Graphite is an amazingly powerful tool from Orbitz for visualizing metrics, but there is a lack of tools for gathering host-level stats and sending into graphite.
It would be great if we could leverage the strengths of both tools.
Graylog2 is an open-source self-hosted centralized log management tool. Think of it as a do-it-yourself version of loggly.com, or perhaps a simpler alternative to Splunk. Logs are stored in a MongoDB database.
I recently had a need to upload a few large (11GB) files to S3 for offsite archive purposes. Amazon announced support for objects >5GB back in November 2010, so I had assumed that most S3 clients and libraries had support for it already. I was wrong. It turns out that in order to support large files, you need to use the new “multipart upload” feature. What I found was that many S3 clients do not yet support multipart uploads.
In case this helps save anyone else a few hours of time, here is a summary of what I found regarding multipart upload support in the various S3 clients:
Back in late 2010 I had a need to create a private Cloudfront streaming distribution but there was no simple way to do this. Amazon’s web-based AWS Management Console did not support this and I could not find any simple CLI tools for doing this.
Luckily, RightScale’s right_aws ruby gem (>= 2.0.0) provides support for the Cloudfront private streaming API.
These tools should make it simple for any linux admin or developer to get started with creating a private streaming distribution on Amazon.
Sometimes it might be nice to be able to test a DNS server’s output, such as with a continuous monitoring system, or as a validation tool when migrating to a different DNS server (or service.)
$ dns_compare.py -z example.com --file example.com.zone --server 10.1.1.1
....X...X..done
Results:
Matches: 9
Mis-matches: 2
To see more examples or download the code, goto: https://github.com/joemiller/dns_compare
Gitosis-ng – it’s gitosis with some new features to help users work with the git server. Mainly implemented with commands sent via ssh, ex:
Use mc-shellcmd to run any command on the nodes in your collective, eg:
Sometimes while testing you may want to be able to simulate network latency, or packet loss, or low bandwidth. I have done this with Linux and tc/netem as well as with Shunra on Windows, but I had never done it on Mac OSX.
It turns out that Mac OSX includes ‘dummynet’ from FreeBSD which has the capability to do this WAN simulation.
I have been a big fan of TunnelBroker.net’s IPv6 tunneling service for a number of years now, but I was pleasantly surprised to discover a new service from TunnelBroker.net (HE - Hurrican Electric) – IPv6 DNS hosting! Actually it’s dual-stack DNS hosting, allowing you to serve DNS requests for your domains to both the IPv4 and IPv6 internet.
It is currently in “open beta” status and available to anyone with a (free) ipv6 tunnelbroker.net account, or if you happen to be a customer of the parent company providing this service, HE (Hurricane Electric.)
The BIND-DLZ project publishes an excellent set of performance testing toolswhich make it easy to generate a lot of fake DNS data for a variety of DNS server types.
I have extended this excellent tool set by creating a few new “writer” modules:
- contrib::writers::mydns::file
- contrib::writers::powerdns::xdb
- contrib::writers::powerdns::sqlite3
- contrib::writers::powerdns::gmysql
- contrib::writers::windowsdns::file
This is a module for php4 that I wrote a very long time ago that provides access to the rrdtool C library to php scripts.