To paraphrase a civil war general:
"The only good code is deleted code."
I have recently been in several meetings where the phrase
"We'll save resources by doing things with new software",
has been used a justification for change.
This is one of the great lies that drives Silicon Valley. Every line of software is a cost liability.
The only way to save money with software is to delete the number of lines of software you run or to use software to make the problem somebody else's problem.
Every advance in computer software engineering has followed these two principals.
Compiled languages follow this principal by both
reducing the number of lines of code required to solve a problem and also by handing the problem of turning those lines of code into executable instructions to the people that write compilers.
Operating systems follow the same principal; instead of having to manage
access to millions of blocks in both memory and disk, you write enough software to make it the user's problem.
The only way to save resources with new software is to delete old software, or to make the things that the old software was responsible for somebody else's problem.
New software can make doing new things cheaper, but that is a completely different problem.
Thursday, May 23, 2013
Wednesday, May 22, 2013
How to build ruby 1.9.3 with libyaml in a funky place
The only solution I've found is to hack the location of libyaml library and include files
into the extconf.rb file in ext/psych
into the extconf.rb file in ext/psych
header_dir = '/afs/slac/package/ruby/@sys/include'
library_dir = '/afs/slac/package/ruby/@sys/lib'
dir_config 'libyaml', header_dir, library_dir
Friday, October 12, 2012
Why Monitoring Sucks
It doesn't matter what software you use, the fundamental model behind all current monitoring software is broken.
The idea the originally prompted me to create this blog still holds true, I never published it. But the basic tenent of that message still holds true.
"Push Considered Harmful"
The original idea I had was in the context of grid computing in the early 2000's. Early versions of the LHCG were based on the idea that a central scheduler could monitor the entire state of the "grid" and
assign jobs to the hosts that had both the data and the available cycles. This failed badly, and the concept of "pilot jobs" invaded LHCG. In essense, pilot jobs flip the paradigm on it's head and turn "push" architectures into "pull" ones. Instead of a node passive waiting for the central service to ask
it for state, the central service should be waiting for nodes to notify it for state.
There have been no large scale push architectures that have actually ever worked outside a single organization. This is an important lesson for monitoring.
The reason "monitoring sucks" is that it is entirely based on a push model. All current monitoring software assumes that you have a model of your infrastructure that can be mapped onto existing
hardware either virtual or real. This is completely backwards.
There will be no improvement in monitoring software until this model is thrown out and a "pull" or
peer to peer model is created. There is a reason Nagios still exists, it's because everything else that
has tried to replace it uses the same "push" model.
What is needed is a subscribe model in which a node requests a monitoring service to "pull" service requests with a finite time limit. Similarly there needs to be a subscribe service for alerts. Monitoring configuration needs to be a dynamic service; any software that reads a static file for configuration is
doomed to suck.
The other fundamental problem in monitoring is the poor separation between alert and diagnostic services. For diagnostics, you want to monitor "everything", for alerts you should only be monitoring
things that you can actually control. If it sends an alert, there needs to be procedure to handle the alert.
Until monitoring services look a lot more like DHCP than DNS, they will continue to suck, regardless of the software used to implement them.
The idea the originally prompted me to create this blog still holds true, I never published it. But the basic tenent of that message still holds true.
"Push Considered Harmful"
The original idea I had was in the context of grid computing in the early 2000's. Early versions of the LHCG were based on the idea that a central scheduler could monitor the entire state of the "grid" and
assign jobs to the hosts that had both the data and the available cycles. This failed badly, and the concept of "pilot jobs" invaded LHCG. In essense, pilot jobs flip the paradigm on it's head and turn "push" architectures into "pull" ones. Instead of a node passive waiting for the central service to ask
it for state, the central service should be waiting for nodes to notify it for state.
There have been no large scale push architectures that have actually ever worked outside a single organization. This is an important lesson for monitoring.
The reason "monitoring sucks" is that it is entirely based on a push model. All current monitoring software assumes that you have a model of your infrastructure that can be mapped onto existing
hardware either virtual or real. This is completely backwards.
There will be no improvement in monitoring software until this model is thrown out and a "pull" or
peer to peer model is created. There is a reason Nagios still exists, it's because everything else that
has tried to replace it uses the same "push" model.
What is needed is a subscribe model in which a node requests a monitoring service to "pull" service requests with a finite time limit. Similarly there needs to be a subscribe service for alerts. Monitoring configuration needs to be a dynamic service; any software that reads a static file for configuration is
doomed to suck.
The other fundamental problem in monitoring is the poor separation between alert and diagnostic services. For diagnostics, you want to monitor "everything", for alerts you should only be monitoring
things that you can actually control. If it sends an alert, there needs to be procedure to handle the alert.
Until monitoring services look a lot more like DHCP than DNS, they will continue to suck, regardless of the software used to implement them.
Friday, May 20, 2011
On the Eve of the Fakapocalypse
While scrolling through the channel guide I came across this gem.
And it reminded me that I was born into a world that really was 30 seconds from the apocalypse. We can laugh now at the fakapocalypse of the bad math of deluded visionaries, but not that long ago the world really was at the brink of total destruction.
We still have the capability to turn the world into an ash cinder, but we seem to have run out of enemies willing to put that much on the table.
Maybe we still are doomed as a species, after all we have not escaped the brutal equations of Malthus. Finite resources and exponential population growth have only one outcome. I find some hope in the fact that more than cock roaches and deep sea tube worms might survive our extinction.
But still, no matter how bleak our world view gets, it seems like there will be a world to view. That's a lot more than most people thought possible when I was born.
- Booker C. Bense
Subscribe to:
Posts (Atom)