tag:blogger.com,1999:blog-80309775444826074062024-03-13T12:07:36.578-07:00Curse the DarknessRants about the world.Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.comBlogger24125tag:blogger.com,1999:blog-8030977544482607406.post-12251269313201812542023-09-19T23:04:00.001-07:002023-09-19T23:14:17.270-07:00Why the Mai Tai is the Perfect Mirror Cocktail<p> The Mai Tai is The Perfect Mirror Cocktail</p><p>The Mai Tai enjoys a special place in the history of cocktails in that it has just enough history that it almost can be itemized and specified to a point in time. Yet, none of those original ingredients can be currently obtained, and the actual cocktail that gained world wide fame only loosely resembles what we know of the origins of the Mai Tai. </p><p>Thus the Mai Tai is a holy grail that can not be recreated. It's only 4 ingredients, yet each one of them can become a quest to recreate something that does not exist in the modern world. This is the magic of the Mai Tai. It becomes a mirror of the person recreating the drink; what do they think matters? Most efforts have focused on recreating the unavailable 17 year old Wrey & Nephew Jamaican rum, but what orange liquor was used in the "original" Mai Tai? Or for that matter, what lime variety was used for the juice? What exactly was the "rock candy" syrup? I won't even attempt to plumb the depths of what "orgeat" was in the 40's. </p><p>This creates a perfect mirror for allowing the mixologist to express their vision of what a Mai Tia is within a limited framework. Much like the limited poetic frameworks of haiku or sonnets, restrictions focus creativity. 400 years after Basho, people are still attempting to recreate his simple formula in completely different languages to describe the natural world. And just like the Mai Tai, we can never completely know the subtleness of 16th Century Japanese. I fully expect that in 400 years, mixologists will be attempting to recreate the Mai Tai in whatever ingredients exist. It might be better, it might be worse, it will likely taste nothing like whatever was served in Oakland in the 40's. </p><p>To be honest, I suspect many of the current recreations of the Mai Tai are much better than whatever you think of as the original, but there is no way to know. This is the beauty and wonder of the Mai Tai. It is a mirror in which you express yourself in a common language. </p>Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-51968314412670158612015-10-30T20:20:00.000-07:002015-10-30T20:35:09.203-07:00The Definitive All Dancing, All Complete, "WTF Happened to my nice list of integers in Elixir?" Post. This question happens over and over on every forum where people just starting out with Elixir<br />
go to ask questions. Especially with folks that start with the Euler challenges as a way to learn<br />
Elixir. It generally starts like this.<br />
<br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> Enum.map(1..5, fn(x) -> x*x end )</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">[1, 4, 9, 16, 25]</span><br />
<span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">iex> Enum.map(6..10, fn(x) -> x*x end )</span></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">'$1@Qd'</span><br />
<br />
And the reaction is always more or less...<br />
<br />
<span style="color: red;"><b><i>"WTF! where did my nice list of integers go?</i></b>"</span><br />
<br />
What is really happening is something that is generally so hidden for users of<br />
languages like Ruby or Python is that they've forgotten it exists. Computers do<br />
not store data as readable sections of text, but as binary sequences of ones and<br />
zeros. Ruby et al, do a lot of work to turn those sequences of binary data into<br />
readable strings that can be printed at a terminal. Elixir does the same, it<br />
uses the Inspect Protocol to turn every result that is returned by Elixir<br />
expressions into a readable string.<br />
<br />
So lets look at the code that actually gets run when you type an Elixir expression in IEx that returns a List.<br />
<br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;">defimpl Inspect, for: List do</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> def inspect([], _opts), do: "[]"</span><br />
<span style="color: orange; font-size: x-small;"><span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;"> def inspect(thing, %Inspect.Opts{char_lists: lists} = opts) do</span></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> cond do</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> lists == :as_char_lists or (lists == :infer and printable?(thing)) -></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> << ?', Inspect.BitString.escape(IO.chardata_to_string(thing), ?') :: binary, ?' >></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> keyword?(thing) -></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> surround_many("[", thing, "]", opts, &keyword/2)</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> true -></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> surround_many("[", thing, "]", opts, &to_doc/2)</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> end</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace; font-size: x-small;"> end</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: inherit;">Now, the key thing understand here is that the default for the Inspect Protocol is</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: inherit;"></span><span style="color: orange; font-family: Courier New, Courier, monospace;">char_lists: infer</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: inherit;">This means that the code will call the </span><span style="font-family: Courier New, Courier, monospace;">printable?</span><span style="font-family: inherit;"> function on the list. If every integer in the </span><br />
<span style="font-family: inherit;">list corresponds to a printable ASCII character (i.e. is in the range </span> 32..126 plus some integers that<br />
represent various whitespace and newline characters. ), it will print the list as a string rather than<br />
that nice list of integers separated by commas and surrounded by brackets that you were expecting.<br />
<br />
Now you are asking yourself:<br />
<br />
<b><i><span style="color: red;">"Why does Elixir do this batshit transformation just because my list is small numbers?"</span></i></b><br />
<b><i><br /></i></b>
It's a good question to ask. Elixir has a perfectly fine String data type that doesn't look anything like<br />
a List of small numbers. The reason Elixir has this logic in it's List Inspect implementation is that Elixir is built on top of Erlang and all Erlang functions are perfectly valid Elixir functions. This is<br />
one of the great strengths of Elixir. Elixir users get 20+ years of solid software engineering that is used to manage a large percentage of the world's cell phone traffic.<br />
<br />
Unfortunately, this comes with a price. The price is the list to string transformation above. You see in Erlang, strings are implemented in what was the style at the time:<br />
<br />
<h4>
<b><span style="color: red;">Lists of Integers </span></b></h4>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://stream1.gifsoup.com/view3/4484271/grandpa-simpson-potato-dance-o.gif" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://stream1.gifsoup.com/view3/4484271/grandpa-simpson-potato-dance-o.gif" /></a></div>
<br />
<br />
This means that if the Inspect Protocol has tell you about Erlang error messages or strings returned from Erlang functions, it must convert any List of small integers into an ASCII string. Well, lets see what happens if we turn that transformation off.<br />
<br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex(15)> :application.info</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">[loaded: [{:iex, 'iex', '1.1.1'}, {:stdlib, 'ERTS CXC 138 10', '2.6'},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:logger, 'logger', '1.1.1'}, {:compiler, 'ERTS CXC 138 10', '6.0.1'},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:elixir, 'elixir', '1.1.1'}, {:kernel, 'ERTS CXC 138 10', '4.1'}],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> loading: [],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> started: [logger: :temporary, iex: :temporary, elixir: :temporary,</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> compiler: :temporary, stdlib: :permanent, kernel: :permanent],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> start_p_false: [],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> running: [logger: #PID<0.48.0>, iex: #PID<0.42.0>, elixir: #PID<0.35.0>,</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> compiler: :undefined, stdlib: :undefined, kernel: #PID<0.9.0>], starting: []]</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: inherit;">That's all very useful information. There are many Erlang functions that simply don't have Elixir </span><span style="font-family: inherit;">aware wrappers to do the transformation from Erlang to Elixir strings. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Now if we turn off the nasty</span><span style="font-family: Courier New, Courier, monospace;"> :infer </span><span style="font-family: inherit;">that destroyed our beautiful list of integers, what </span><span style="font-family: inherit;">happens to all that useful information. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> IEx.configure inspect: [ char_lists: false]</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">:ok</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> :application.info</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">[loaded: [{:iex, [105, 101, 120], [49, 46, 49, 46, 49]},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:stdlib, [69, 82, 84, 83, 32, 32, 67, 88, 67, 32, 49, 51, 56, 32, 49, 48],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> [50, 46, 54]},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:logger, [108, 111, 103, 103, 101, 114], [49, 46, 49, 46, 49]},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:compiler, [69, 82, 84, 83, 32, 32, 67, 88, 67, 32, 49, 51, 56, 32, 49, 48],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> [54, 46, 48, 46, 49]},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:elixir, [101, 108, 105, 120, 105, 114], [49, 46, 49, 46, 49]},</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:kernel, [69, 82, 84, 83, 32, 32, 67, 88, 67, 32, 49, 51, 56, 32, 49, 48],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> [52, 46, 49]}], loading: [],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> started: [logger: :temporary, iex: :temporary, elixir: :temporary,</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> compiler: :temporary, stdlib: :permanent, kernel: :permanent],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> start_p_false: [],</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> running: [logger: #PID<0.48.0>, iex: #PID<0.42.0>, elixir: #PID<0.35.0>,</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> compiler: :undefined, stdlib: :undefined, kernel: #PID<0.9.0>], starting: []]</span><br />
<br />
<span style="font-family: inherit;">That's terrible, what do all those weird lists of number mean? Now imagine you got an error message </span><span style="font-family: inherit;">that was just a list of integers. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">It really seems like there should be a way to straighten this out, but unfortunately all the solutions are worse than the problem. However, once Elixir Golf becomes a thing ( and it's just a matter of time) , </span><span style="font-family: inherit;">you will be able to turn that nastily long list of integers into a cryptic string and your code will still </span><span style="font-family: inherit;">work. </span><br />
<span style="font-family: inherit;"><br /></span>
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> Enum.map( '%*&^%*&^%' , fn(x) -> x*x end )</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;">[1369, 1764, 1444, 8836, 1369, 1764, 1444, 8836, 1369]</span><br />
<br />
<br />
And finally for those that have made it this far, some real dancing.<br />
<iframe allowfullscreen="" class="giphy-embed" frameborder="0" height="202" src="//giphy.com/embed/143I6D1bXdXPyw" width="480"></iframe><br />
<a href="http://giphy.com/gifs/pulp-fiction-uma-thurman-143I6D1bXdXPyw">via GIPHY</a><br />
<br />
<br />Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-69864928446495387072015-08-28T13:21:00.000-07:002015-08-28T13:21:33.656-07:00Making Complex Streams Easier to Understand<span style="font-family: Times, Times New Roman, serif;">Recently the topic of complicated pipe sequences came up on the Elixir Slack channel and there were comments about the inscrutability of a really complicated Stream built of pipes. This is a typical example:</span>
<br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<br />
<pre> <span style="color: orange; font-family: Courier New, Courier, monospace;">File.stream!</span><span style="color: orange; font-family: 'Courier New', Courier, monospace;">("priv/data/stops.csv")</span><span style="color: orange; font-family: Courier New, Courier, monospace;">
|> Stream.drop(1)
|> Stream.map</span><span style="color: orange; font-family: Courier New, Courier, monospace;">(</span><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">(&(Station.fromCSVRow &1)</span></span><span style="color: orange; font-family: Courier New, Courier, monospace;">
|> Enum.each(&(Station.Store.put(&1)))</span></pre>
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="font-family: Times, Times New Roman, serif;">One thing to always keep in mind is that Elixir is an expression based language and any result can always be replaced by a function. The other is that Streams are merely a composition of functions. </span>
<span style="font-family: Times, Times New Roman, serif;">A more readable way to write the above would be </span>
<br />
<pre><span style="color: orange; font-family: Courier New, Courier, monospace;"> </span></pre>
<pre><span style="color: orange; font-family: Courier New, Courier, monospace;">def station_stream(file) do</span></pre>
<pre><span style="color: orange; font-family: Courier New, Courier, monospace;"> File.stream!(file) </span></pre>
<pre><span style="color: orange; font-family: Courier New, Courier, monospace;"> |> </span><span style="color: orange; font-family: 'Courier New', Courier, monospace;">Stream.drop(1)</span></pre>
<pre><span style="color: orange; font-family: 'Courier New', Courier, monospace;"> |> </span><span style="color: orange; font-family: Courier New, Courier, monospace;">Stream.map(</span><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">(&(Station.fromCSVRow &1))</span></span></pre>
<pre><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">end</span></span></pre>
<pre><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">
</span></span></pre>
<pre><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">station_stream(</span></span><span style="color: orange; font-family: 'Courier New', Courier, monospace;">"priv/data/stops.csv")</span></pre>
<pre><span style="color: orange; font-family: 'Courier New', Courier, monospace;"> |> Enum.</span><span style="color: orange; font-family: 'Courier New', Courier, monospace;">each(&(Station.Store.put(&1))</span></pre>
<pre><span style="color: orange; font-family: 'Courier New', Courier, monospace;">
</span></pre>
<span style="font-family: Times, Times New Roman, serif;">Additionally, you can put that Stream in a variable and use it anywhere in the program.
As long as the underlying file does not change, the values of the enumeration will not change. </span>
<br />
<pre><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">
</span></span></pre>
<pre><span style="color: orange;"><span style="font-family: Courier New, Courier, monospace;">stops = station_stream(</span></span><span style="color: orange; font-family: 'Courier New', Courier, monospace;">"priv/data/stops.csv")</span>
<span style="font-family: Times, Times New Roman, serif;">A File.stream always enumerates through the whole file each time it is evaluated. </span>
<span style="font-family: Times, Times New Roman, serif;">Pipes are one of the features that draws programmers to Elixir, but like anything good you can always over do it. Readability and maintainability should be the first goal. I'm not sure where the exact limit is, but I am sure that pipes can be abused to make code very difficult to reason about. My personal limit seems to be about 3-4 in a sequence.</span></pre>
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-81783444181029296652015-08-01T08:39:00.000-07:002015-08-01T08:39:36.379-07:00Thinking about Stream and Enum in Terms of Function CompositionComing from a Ruby background it took me a while to wrap my head around what<span style="font-family: Courier New, Courier, monospace;"> Stream</span> and <span style="font-family: Courier New, Courier, monospace;">Enum </span>really are. It's far too easy to think about them as a generalized version of a Ruby Array.<br />
<br />
At their heart, they are both tools for composing functions.<br />
<br />
Let's start with Enum, in most object style languages enumerators support an "each" method<br />
that allows you to operate one at a time on members of the collection.<br />
<br />
Elixir <span style="font-family: Courier New, Courier, monospace;">Enum</span> looks quite different at first. It requires that the collection implement these basic functions.<br />
<br />
<section class="details_list" id="types_details" style="-webkit-text-stroke-width: 0px; color: black; display: block; font-family: 'Lucida Sans', 'Lucida Grande', Verdana, Arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: 21px; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;"></section><br /><br />
<table class="summary" style="-webkit-text-stroke-width: 0px; border-collapse: separate; border-spacing: 0px; border: 0px; font-family: 'Lucida Sans', 'Lucida Grande', Verdana, Arial, sans-serif; letter-spacing: normal; margin-top: 1em; orphans: auto; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px;"><tbody>
<tr style="background-color: #f0f0f0; background-position: initial initial; background-repeat: initial initial;"><td class="summary_signature" style="border: 0px; padding: 0.5em 0.5em 0.5em 0.4em;"><a href="file:///Users/booker/Library/Application%20Support/Dash/DocSets/Elixir/Elixir.docset/Contents/Resources/Documents/docs/elixir/Enumerable.html#count/1" style="background-color: transparent; color: #0055aa; text-decoration: none;">count(collection)</a></td><td class="summary_synopsis" style="border: 0px; padding: 0.5em 1em 0.5em 0.5em;"><div style="margin: 0px;">
Retrieves the collection’s size</div>
</td></tr>
<tr style="background-color: #fafafa; background-position: initial initial; background-repeat: initial initial;"><td class="summary_signature" style="border: 0px; padding: 0.5em 0.5em 0.5em 0.4em;"><a href="file:///Users/booker/Library/Application%20Support/Dash/DocSets/Elixir/Elixir.docset/Contents/Resources/Documents/docs/elixir/Enumerable.html#member?/2" style="background-color: transparent; color: #0055aa; text-decoration: none;">member?(collection, value)</a></td><td class="summary_synopsis" style="border: 0px; padding: 0.5em 1em 0.5em 0.5em;"><div style="margin: 0px;">
Checks if a value exists within the collection</div>
</td></tr>
<tr style="background-color: #f0f0f0; background-position: initial initial; background-repeat: initial initial;"><td class="summary_signature" style="border: 0px; padding: 0.5em 0.5em 0.5em 0.4em;"><a href="file:///Users/booker/Library/Application%20Support/Dash/DocSets/Elixir/Elixir.docset/Contents/Resources/Documents/docs/elixir/Enumerable.html#reduce/3" style="background-color: transparent; color: #0055aa; text-decoration: none;">reduce(collection, acc, fun)</a></td><td class="summary_synopsis" style="border: 0px; padding: 0.5em 1em 0.5em 0.5em;"><div style="margin: 0px;">
Reduces the collection into a value</div>
</td></tr>
</tbody></table>
<br />
<div>
None of these looks like an "each" method[1]. </div>
<div>
<br /></div>
<div>
All the <span style="font-family: Courier New, Courier, monospace;">Enum</span> functions do is use these 3 basic functions to provide handy shortcuts for </div>
<div>
using the reduce function of the original collection. Unlike a typical "each" method in Ruby,</div>
<div>
an <span style="font-family: Courier New, Courier, monospace;">Enum</span> function can bail out at any time. For example, look at the implementation for </div>
<div>
<span style="font-family: Courier New, Courier, monospace;">Enum.all?</span></div>
<div>
<br /></div>
<div>
<div>
<span style="color: orange; font-family: Courier New, Courier, monospace;"> def all?(collection, fun) do</span></div>
<div>
<span style="color: orange; font-family: Courier New, Courier, monospace;"> Enumerable.reduce(collection, {:cont, true}, fn(entry, _) -></span></div>
<div>
<span style="color: orange; font-family: Courier New, Courier, monospace;"> if fun.(entry), do: {:cont, true}, else: {:halt, false}</span></div>
<div>
<span style="color: orange; font-family: Courier New, Courier, monospace;"> end) |> elem(1)</span></div>
<div>
<span style="color: orange; font-family: Courier New, Courier, monospace;"> end</span></div>
</div>
<div>
<br /></div>
<div>
Like most of the functions in <span style="font-family: Courier New, Courier, monospace;">Enum</span>, it's using the collection's reduce function to implement a specific kind of reduction. In effect it's a composition of functions. It's also "lazy" in the sense that it only iterates as far as it has to generate the correct result. It's important to note that Enumerable implements a different reduce function than the standard one in Enum. ( The accumulator variable must return a tuple consisting of status and "real" accumulator. )</div>
<div>
<br /></div>
<div>
Now let's consider the case of <span style="font-family: Courier New, Courier, monospace;">Stream. </span><span style="font-family: Times, Times New Roman, serif;">What it does is very similar to </span><span style="font-family: Courier New, Courier, monospace;">Enum</span><span style="font-family: Times, Times New Roman, serif;">, except that it doesn't</span><br />
<span style="font-family: Times, Times New Roman, serif;">actually execute the reduce function. In effect it creates an anonymous function for transforming the </span><br />
<span style="font-family: Times, Times New Roman, serif;">collection into another collection. You can use this "dynamic" collection anyplace you can use a regular collection. </span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">iex(9)> foo = 1..10 |> Stream.map( fn(x) -> x * 2 end )</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">#Stream<[enum: 1..10, </span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">funs: [#Function<45.113986093/1 in Stream.map/2>]]></span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">iex(10)> foo |> Enum.max</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">20</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">iex(11)> foo |> Enum.min</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">2</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">iex(12)> foo |> Enum.take(1)</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">[2]</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">iex(13)> foo |> Enum.take(3)</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;">[2, 4, 6]</span></div>
<div>
<br /></div>
<div>
The important thing to note from this example is that the a collection defined by <span style="font-family: Courier New, Courier, monospace;">Stream </span><span style="font-family: Times, Times New Roman, serif;">does not imply state. It behaves just like a normal collection and is just as immutable. Every time you use it</span><br />
<span style="font-family: Times, Times New Roman, serif;">it starts at the beginning. The decision about when to use a </span><span style="font-family: Courier New, Courier, monospace;">Stream</span><span style="font-family: Times, Times New Roman, serif;"> or </span><span style="font-family: Courier New, Courier, monospace;">Enum</span><span style="font-family: Times, Times New Roman, serif;"> is should be based on the tradeoffs of storing the collection verses creating it runtime. One difference that is important to note is that</span><span style="font-family: Courier New, Courier, monospace;"> Stream</span><span style="font-family: Times, Times New Roman, serif;"> only supports using </span><span style="font-family: Courier New, Courier, monospace;">reduce</span><span style="font-family: Times, Times New Roman, serif;">, using</span><span style="font-family: Courier New, Courier, monospace;"> count</span><span style="font-family: Times, Times New Roman, serif;"> or</span><span style="font-family: Courier New, Courier, monospace;"> member</span><span style="font-family: Times, Times New Roman, serif;"> directly from </span><span style="font-family: Courier New, Courier, monospace;">Enumerabl</span><span style="font-family: Times, Times New Roman, serif;">e will fail. Looking at the code for </span><span style="font-family: Courier New, Courier, monospace;">Enum.count</span><span style="font-family: Times, Times New Roman, serif;"> we see how you can use </span><span style="font-family: Courier New, Courier, monospace;">reduce</span><span style="font-family: Times, Times New Roman, serif;"> to </span><br />
<span style="font-family: Times, Times New Roman, serif;">emulate these functions. </span><br />
<span style="font-family: Times, Times New Roman, serif;"><br /></span>
<span style="color: orange; font-family: Courier New, Courier, monospace;">def count(collection) do</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> case Enumerable.count(collection) do</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:ok, value} when is_integer(value) -></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> value</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {:error, module} -></span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> module.reduce(collection, {:cont, 0}, fn</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> _, acc -> {:cont, acc + 1}</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> end) |> elem(1)</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> end</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> end</span></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
[1]- It's simple enough to create an "each" from a reduce once you've played around with enough</div>
<div>
examples of reduce. </div>
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-22420689747011895142015-07-16T16:52:00.001-07:002015-07-16T16:52:46.348-07:00Tilting at Windmills: Accessing Erlang man pages from Iex, Part 2:Digging into the source code of iex, you find that the helpers are macros that pull apart the code and do introspection on the Elixir module to find the documentation. To get the documentation for a specific function in a module the code that eventually gets run is this:<br />
<br />
<span style="color: red; font-family: "Courier New",Courier,monospace;">iex> Code.get_docs(Atom, :docs)</span><br />
<span style="color: red; font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">[{{:to_char_list, 1}, 22, :def, [{:atom, [], nil}],</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"> "Converts an atom to a char list.\n\nInlined by the compiler.\n"},</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"> {{:to_string, 1}, 12, :def, [{:atom, [], nil}],</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"> "Converts an atom to string.\n\nInlined by the compiler.\n"}]</span><br />
<br />
Where module is a Atom. All module names are Atoms. Code.get_docs returns a list of tuples describing the attributes of each of the functions exported by the module. <br />
<br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"> {{function, arity }, line, kind, signature, text}</span><br />
<span style="color: orange; font-family: Courier New, Courier, monospace;"><br /></span>
The initial tuple is the function name and arity. line is the line of the source file on which the function. kind is an atom that describes whether the exported function is a macro or a function.<br />
signature is a list of tuples that define the arguments that the function takes, and lastly text is<br />
the markdown documentation for the function.<br />
<br />
If you're willing to fudge things a bit you can duplicate the parts of this function that the iex h command uses by parsing erlang man pages. Every erlang module supports the function module_info.<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"><span style="color: red;">iex> Atom.module_info(:exports)</span></span><br />
<span style="color: orange; font-family: 'Courier New', Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">[__info__: 1, to_string: 1, to_char_list: 1, module_info: 0, module_info: 1]</span><br />
<br />
<span style="color: red; font-family: Courier New, Courier, monospace;">iex> :et.module_info(:exports) </span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"><br /></span>
<span style="color: red; font-family: Courier New, Courier, monospace;">[trace_me: 4, trace_me: 5, phone_home: 4, phone_home: 5, report_event: 4,</span><br />
<span style="color: red; font-family: Courier New, Courier, monospace;"> report_event: 5, module_info: 0, module_info: 1]</span><br />
<br />
Since the erlang man pages are created by a standard xml -> nroff process, it's relatively straightforward to split the nroff man page into documentation sections and to identify the function<br />
documented by that section. So creating an ErlangCode.get_docs is possible by mapping the text<br />
from the man page to the appropriate tuple in the export list. You can find code that does this in<br />
the Erlman project on github.<br />
<br />
<a href="https://github.com/bbense/erlman" target="_blank">Erlman</a><br />
<div>
<br /></div>
Unfortunately, this is kind of a dead end since it's not general enough to be included in the main elixir source code. It requires that the erlang man pages be installed and that is simply something that can not be counted on. It's also unclear if the current version will even work on Windows with the man pages installed.<br />
<br />
My next thought is "Is there a hook into iex commands that I can use to make this an optional addon?"Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-26531626323851934942015-06-19T14:40:00.001-07:002015-06-20T10:07:28.529-07:00Tilting At Windmills, Accessing Erlang man pages from Iex Part: 1In the process of learning how to get a cryptographic hash of a file in Elixir, I found myself switching back and forth from<span style="font-family: "Courier New",Courier,monospace;"> iex</span> to <span style="font-family: "Courier New",Courier,monospace;">erl -man</span> to better understand how<br />
to use the available Erlang functions in Elixir. After all there are over 500 modules in the standard Erlang libraries that are available in Elixir. Wouldn't it be nice I thought to just be able to type<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><span style="color: red;">iex> h :crypto.hash</span> </span><br />
<br />
and get some basic information about the functions. Elixir already supports listing all the functions available in both Elixir and Erlang modules, using
the <span style="font-family: "Courier New",Courier,monospace;">module.TAB</span> sequence<br />
<br />
<span style="color: red;"><span style="font-family: Courier New, Courier, monospace;">iex> h :crypto.TAB </span></span><br />
<span style="font-family: Courier New, Courier, monospace;"> </span>
<br />
If we have Erlang installed, the man pages must be there somewhere. And of course one of the tenants of Iex is that any functionality should be available in the Windows version if at all possible. While creating a command to shell out to<span style="font-family: "Courier New",Courier,monospace;"> erl -man</span> would be simple, it wouldn't be portable.<br />
<br />
Erlang installs it's man pages in a separate path to avoid conflicts with the standard unix man
pages but puts them in the standard <span style="font-family: Courier New, Courier, monospace;">man/man3</span> locations. The convention is that the
man page for an Erlang module is the Erlang module name followed by section number. The
standard Erlang functions are documented in the <span style="font-family: Courier New, Courier, monospace;">erlang.3</span> man page.<br />
<br />
So my next step was to reimplement <span style="font-family: "Courier New",Courier,monospace;">erl -man</span> in purely Elixir code. The first part went really
quickly, finding the man page for a given module was relatively straight forward. We just find
where the erl executable is and work backwards from that to find where the erlang man pages
are and then search for the module name.<br />
<br />
<span style="font-size: x-small;"><span style="color: cyan;"><span style="font-family: "Courier New",Courier,monospace;"> <span style="color: #f6b26b;">def manpath do<br /> start = System.find_executable("erl") <br /> case start do <br /> nil -> nil <br /> _ -> find_manpath(start) <br /> end <br /> end<br /><br /> defp find_manpath(erl_path) do<br /> mpath = erl_path |><br /> Path.split |> <br /> Stream.scan(&Path.join(&2,&1)) |> <br /> Enum.filter( fn(p) -> File.exists?(Path.join([p,"man","man3","ets.3"])) end ) |><br /> List.last<br /> if mpath , do: Path.join(mpath,"man"), else: nil <br /> end </span></span></span></span><br />
<br />
After that I implemented a very simple-minded man nroff macro to markdown translator and was able
to display an Erlang man page from the iex prompt. During this process I noticed that the Erlang
man pages use a very consistant, small subset of the man nroff macro package. This lead me to believe
that it would be possible to extract the specific function documentation from the man pages.<br />
<br />
It turns out that Erlang documentation is maintained in XML and translated by an xmerl based program
into both man pages and html documentation. Unfortunately, this XML source is not included in the default
binary distributions of Erlang. Unix versions get the man pages ( and html usually ), but the Windows
distributions only get the html. Parsing the html is a lot more complicated that dealing with the nroff
and there are no HTML parsers in the standard Elixir libs.<br />
<br />
At this point, I decided to dig a bit deeper and figure out just how<br />
<br />
<span style="font-family: Courier New, Courier, monospace;"><span style="color: red;">iex> h Atom.to_char</span> </span>
<br />
<br />
really works.
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-75692780183697986732015-06-05T10:33:00.000-07:002015-06-12T08:09:13.640-07:00Micro Benchmarking in Elixir using BenchfellaWhen I finished the last post, I had complete intentions of writing a post about how to use the Ports module in Elixir to interaction with the system shell to do I/O redirection. However, I now understand why people complain about Erlang documentation. Ports in Elixir aren't documented other than by a reference to the underlying Erlang libraries and the documentation in Erlang for ports is very incomplete. ( For example, what are the default drivers available in erlang?)<br />
<br />
Ports are gateways between the Erlang VM and external codes and processes. The Erlang VM is sensitive to code "hanging" in any fashion and Ports need drivers that are aware of how to interact with the Erlang VM safely. However I was unable to find any clear documentation of just what drivers are available and the examples I found were inconsistent.<br />
<br />
<h4>
UPDATE: </h4>
<div>
The types and drivers are documented in the man page for erlang in the open_port section. </div>
<div>
<br /></div>
<div>
<span style="color: red; font-family: Courier New, Courier, monospace;">erl -man erlang</span></div>
<br />
<br />
I did however find the <a href="https://github.com/alco/porcelain">Porcelain</a> Elixir module that is both well documented and very straightforward to use. Having taken this long to get back to this, I'd just as soon go on to the actual benchmarks.<br />
<br />
<a href="https://github.com/alco/benchfella" target="_blank">Benchfella </a>is a micro benchmarking framework that works much like ExUnit does for testing and<br />
you can use the same Dave Thomas hack for creating many tests that iterate over a list of values.<br />
<br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;">defmodule Hash do</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> use Benchfella</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> @lengths [1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,16777216]</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> for chunk <- @lengths do </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> <span class="Apple-tab-span" style="white-space: pre;"> </span>@chunk chunk </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> bench "Hash 2**24 file by #{Integer.to_string(@chunk)}" do</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> hash_test("./bench/data_2_24",@chunk) </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> for chunk <- @lengths do </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> @chunk chunk </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> bench "Hash 2**26 file by #{Integer.to_string(@chunk)}" do</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> hash_test("./bench/data_2_26",@chunk)</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> for chunk <- @lengths do </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> @chunk chunk </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> bench "Hash 2**28 file by #{Integer.to_string(@chunk)}" do</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> hash_test("./bench/data_2_28",@chunk)</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> def hash_test(file,chunk) do</span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> File.stream!(file,[],chunk) </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> |> </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end ) </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> |> </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> :crypto.hash_final </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> |> </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> Base.encode16 </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"> end </span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;">end</span><br />
<br />
Benchfella runs each test for as many times as possible in a given interval ( the default is one second ) and returns the average time per test over that interval. Data from each run is stored on the filesystem so you can do comparisons between runs. The plot below shows the results from using a chunk size in powers of 2 from 2**10 to 2**24 to hash a file of size 2**24.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-YNWNfhbzcMA/VXHaBjhLQNI/AAAAAAAACC0/YIw2aiiVKmQ/s1600/hash_plot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="240" src="http://4.bp.blogspot.com/-YNWNfhbzcMA/VXHaBjhLQNI/AAAAAAAACC0/YIw2aiiVKmQ/s320/hash_plot.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
The results are similar for files of size 2**26 and 2**28. As you can see there is a significant advantage to using a large chunk size. ( With an odd bump at 2**23 ) This test<br />
was done on a MacBook Pro with 16gig of memory and an SSD disk drive.<br />
<br />
This shows that using large binaries in Elixir ( and Erlang ) is generally the fastest way to deal with large data sets. Of course you need to make the tradeoff between total available memory and the number of binaries you want to process at a time.<br />
<br />
The other benchmark I tested was to compare using the "chunk" method of hashing the file with a chunk size larger than the file and simply reading the entire file into a string and<br />
computing it's hash. The simple read method was consistently twice as fast as the single chunk method. <br />
<br />
So for my resulting application I choose to pick a chunk size that allows the code to process multiple files at a time and chooses a method for computing the hash based on the<br />
size of the file. Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-62445664686120524342015-05-14T08:33:00.000-07:002015-05-14T08:33:43.731-07:00Benchmarking Elixir File Hashing Step 1. Creating test dataIn my previous post I outlined how to get a cryptographic hash of a file using Elixir. For large files, you want to chunk the file rather than read the whole thing into memory, it would be good to determine if there is an optimal chunk size to use to for passing into the encryption functions.
<br />
<blockquote class="tr_bq">
<span style="color: yellow; font-family: Courier New, Courier, monospace;">iex> File.stream!("./known_hosts.txt",[],2048)<br />
|> Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end )<br />
|> :crypto.hash_final |> Base.encode16
"97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" </span></blockquote>
<br />
The first step in doing this is to create test data files of a specific length. An obvious approach
is to simply open /dev/random and read until you've got enough sample data. A simple shell example.
<br />
<blockquote class="tr_bq">
<span style="color: yellow; font-family: Courier New, Courier, monospace;">head -c 1024 < /dev/random > test.data </span></blockquote>
<br />
Translating that to Elixir looks like
<br />
<blockquote class="tr_bq">
<span style="color: yellow; font-family: Courier New, Courier, monospace;">
iex(1)> File.stream!("/dev/random",[],1024) |> Enum.take(1)
</span><span style="color: red; font-family: Courier New, Courier, monospace;">** (File.Error) could not stream /dev/random: illegal operation on a directory
(elixir) lib/file/stream.ex:81: anonymous fn/2 in Enumerable.File.Stream.reduce/3
(elixir) lib/stream.ex:1012: anonymous fn/5 in Stream.resource/3
(elixir) lib/enum.ex:1740: Enum.take/2</span>
</blockquote>
That sure looks like a bug, /dev/random is not a directory, but a char special device file. While the error message is misleading, there is no actual bug. Erlang will not open files for reading that it considers dangerous to the overall
scheduler. In this case, /dev/random is a character special device file and since these kinds of files usually block
on I/O, Erlang errs on the side of caution and will refuse to open the file. There is an exception in the Erlang code
for /dev/null since that is considered safe for the scheduler. This post goes into the details.
<br /><br />
<a href="https://azunyanmoe.wordpress.com/2011/03/22/reading-device-files-in-erlang/">Reading Device Files in Erlang</a>
<br /><br />
There are several ways to get around this problem. The first solution that springs to mind is actually
one of the more difficult ones to do in Elixir. In many languages there is a system call that you can use to
execute shell commands. Elixir has System.cmd, but it is relatively limited. You can specify the command to
execute and the argument list, but you cannot use shell based I/O redirection.
<br /><br />
The most straightforward Elixir solution is to use the rand_bytes function from the Erlang crypto library.
<blockquote class="tr_bq">
<span style="color: yellow; font-family: Courier New, Courier, monospace;">
iex(1)> File.write("test.data",:crypto.rand_bytes(1024))
</span>
</blockquote>
<br /><br />
But that isn't much fun, and while it solves this problem it doesn't give us a tool for interacting
with external programs. We'll look at more general solutions in the next post...
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-38636095936678096722015-04-29T14:04:00.000-07:002015-05-13T11:08:03.538-07:00How to get a hash of a file in ExilirAs the next step in learning about elixir I wanted to add a file validator module to my elixgrep project. Of course
this requires as a first step taking the cryptographic hash of a file. I found this <a href="http://www.djm.org.uk/cryptographic-hash-functions-elixir-generating-hex-digests-md5-sha1-sha2/">handy blog entry</a>, but it didn't answer the whole problem.
For small files, you can just read in the whole file into a string and hash it.<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: Courier New, Courier, monospace;"> <span style="color: yellow;">iex> :crypto.hash(:sha256,File.read!("./known_hosts.txt")) |> Base.encode16 "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" </span></span></blockquote>
<br />
But there reaches a point were the file size is large enough that loading the whole
contents in memory isn't performant and in some cases not feasible. My next idea
was to use File.stream!
<br />
<blockquote class="tr_bq">
<span style="color: yellow; font-family: 'Courier New', Courier, monospace;">iex> File.stream!("./known_hosts.txt") |><br />
Enum.reduce(:crypto.hash_init(:sha256),<br />fn(line, acc) -> :crypto.hash_update(acc,line) end ) |><br />
:crypto.hash_final |> Base.encode16<br />
"97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" </span></blockquote>
<br />
However there is still a problem with this in that it assumes the file has appropriate line endings.
For a cryptographic hash it makes more sense to divide up the file into equal byte length chunks.
File.stream!/3 has two default arguements, modes and lines_or_bytes, if you want to stream in
by byte_length use this form.<br />
<br />
<blockquote class="tr_bq">
<span style="color: yellow; font-family: Courier New, Courier, monospace;">iex> File.stream!("./known_hosts.txt",[],2048) |>
Enum.reduce(:crypto.hash_init(:sha256),<br />fn(line, acc) -> :crypto.hash_update(acc,line) end ) |>
:crypto.hash_final |> Base.encode16
"97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" </span></blockquote>
<br />
<br />
Now the interesting question becomes is there an optimal byte size to use for this hashing? STAY TUNED... Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-54303236930703591522015-04-16T13:46:00.001-07:002015-05-13T11:08:24.760-07:00Elixir functions are not FunctionsI was banging my head against a simple elixir test this morning. It's a test that's been in my code for months and that never worked for some reason. This is a boiled down example:<br />
<br />
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline; widows: 1;">
<blockquote style="color: #FFFF00; font-family: Arial, Helvetica, sans-serif; font-size: 13px;">
<br class="Apple-interchange-newline" />
defmodule RaiseTest do<br />
def testraise do<br />
raise "This is an error"<br />
end<br />
end<br />
<br />
defmodule RaiseTestTest do<br />
use ExUnit.Case<br />
test "assert_raise works" do<br />
assert_raise(RuntimeError, RaiseTest.testraise )<br />
end<br />
end<br />
<br />
mix test<br />
<br />
1) test assert_raise works (RaiseTestTest)<br />
test/raise_test_test.exs:4<br />
** (RuntimeError) This is an error<br />
stacktrace:<br />
(raise_test) lib/raise_test.ex:4: RaiseTest.testraise/0<br />
test/raise_test_test.exs:5<br />
<br />
<br />
Finished in 0.03 seconds (0.03s on load, 0.00s on tests)<br />
1 tests, 1 failures</blockquote>
<div>
<span >My error was finally pointed out to me this morning on the elixir list and opened my eyes to a consistent flaw in my thinking about elixir. The fix of course is to actually provide a function to assert_raise, what I was actually providing was an expression that could return anything. </span></div>
<div style="widows: auto;">
</div>
<br />
<div style="widows: auto;">
<blockquote class="tr_bq">
<span style="color: #FFFF00; font-family: Arial, Helvetica, sans-serif; font-size: 13px; widows: 1;"> test "assert_raise works" do<br /></span><span style="color: #FFFF00; font-family: Arial, Helvetica, sans-serif; font-size: 13px; widows: 1;"> assert_raise(RuntimeError, fn -> RaiseTest.testraise end )</span><span style="color: #FFFF00; font-family: Arial, Helvetica, sans-serif; font-size: 13px; widows: 1;"> <br/> end</span></blockquote>
</div>
<div style="widows: auto;">
<span ><br /></span>
<span >The key misunderstanding in my head was that function when used in the elixir context actually maps to what I think of as a function reference from my years of C programming. Module functions return expressions which can be anything. The test failed because it was calling TestRaisetest to see if it returned a function it could use in the test. <br />When an elixir function requires a "function" as an argument, it really means a closure that can be executed.</span></div>
</div>
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-29307371954257620902015-04-03T10:35:00.000-07:002015-04-03T10:35:40.829-07:00Remember JSON is always valid YAMLWhile I don't mind YAML for relatively simple data files, I find it to be very difficult to use when there are more than one or two levels of nesting. If you have a relatively complex config file, then I find JSON to be a much easier to reason about and make any necessary changes.<br />
<br />
For current versions of YAML, JSON files are a subset of the YAML standard. Thus any compliant YAML parser will also parson JSON.<br />
<br />
The place where I use this trick the most often is in .kitchen.yml files used by TestKitchen when writing Chef cookbooks. I start out by taking an existing .kitchen.yml file and converting it to<br />
.kitchen.json using this website<a href="http://codebeautify.org/yaml-to-json-xml-csv" target="_blank"> http://codebeautify.org/yaml-to-json-xml-csv</a>. I then copy the<br />
.kitchen.json to .kitchen.yml.<br />
<br />
<br />Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com1tag:blogger.com,1999:blog-8030977544482607406.post-60842211456482047212015-02-24T11:55:00.003-08:002015-05-13T11:08:47.130-07:00Writing Effective Examples in Elixir docstrings.I had an email exchange on the elixir-core mailing list recently that I think is worth preserving as a blog post. It shows a great approach to how to write examples for functions with complex output.<br />
<br />
<div class="H1SYRSB-wb-W H1SYRSB-wb-Y H1SYRSB-b-Rb H1SYRSB-wb-X" style="border-bottom-color: transparent; border-bottom-style: solid; border-left-color: rgb(77, 144, 240); border-left-style: solid; border-top-color: transparent; border-top-style: solid; border-width: 1px 0px 1px 1px; margin: 0px; outline: none; padding: 4px 0px; position: relative; vertical-align: baseline;" tabindex="0">
<div class="H1SYRSB-wb-W" style="border-bottom-color: transparent; border-bottom-style: solid; border-left-color: transparent; border-left-style: solid; border-top-color: transparent; border-top-style: solid; border-width: 1px 0px 1px 1px; margin: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<div class="H1SYRSB-wb-F" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-wb-x" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-kd-a" style="border: 0px; margin: 0px 0px 0px 44px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<div class="H1SYRSB-wb-P" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;" tabindex="0">
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; margin: 0px; overflow: auto; padding: 0px; vertical-align: baseline;">
<div dir="ltr" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">We had a meetup this weekend for hacking the docs. The intent was to add examples to the standard library. </span><div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">I started looking around in code.ex and the question I run into is the following:</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Should examples be functionally accurate or readable? </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Here's what I mean. I was attempting to add an example for </span></div>
<div style="border: 0px; color: #222222; margin: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;">Code.get_docs. </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;">This is what I came up with. </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;">## Examples</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> iex> Code.get_docs(Atom, :docs)</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> [{{:to_char_list, 1}, 36, :def, [{:atom, [], nil}],</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> "Converts an atom to a char list.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n iex> Atom.to_char_list(:\"An atom\")\\n 'An atom'\\n\\n"},</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> {{:to_string, 1}, 20, :def, [{:atom, [], nil}],</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> "Converts an atom to a string.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n iex> Atom.to_string(:foo)\\n \"foo\"\\n\\n"}]</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> iex(1)> Code.get_docs(Atom, :all )</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> [docs: [{{:to_char_list, 1}, 36, :def, [{:atom, [], nil}],</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> "Converts an atom to a char list.\\n\\\nInlined by the compiler.\\n\\n## Examples\\n\\n iex> Atom.to_char_list(:\"An atom\")\\n 'An atom'\\n\\n"},</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> {{:to_string, 1}, 20, :def, [{:atom, [], nil}],</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> "Converts an atom to a string.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n iex> Atom.to_string(:foo)\\n \"foo\"\\n\\n"}],</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: cyan; font-family: Courier New, Courier, monospace;"> moduledoc: {1,"Convenience functions for working with atoms.\\n\\nSee also `Kernel.is_atom/1`.\\n"}]</span></div>
<div style="border: 0px; color: #222222; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-family: Courier New, Courier, monospace;"> </span></div>
</div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><span style="font-family: Times, Times New Roman, serif;">It's really messy to read and I'm still working on </span>getting the quoting correct for ex_doc. </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Atom is about the simplest standard module available, so picking anything else just </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">gets worse. Also it suffers from the problem that it doesn't automatically update when</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">the docs for Atom update. </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">My initial thought was to dynamically create a Sample module and then use Code.get_docs</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">on that example ( i.e. similar to the test for Code.get_docs ). </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">So I guess the question is: </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Should any examples in the documentation exactly match the results from running the code in iex? </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">i.e. while the standard lib does not use doctest as far as I know, any examples included should be</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">doctest aware. Or is it okay to simplify the results for clarity? </span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">- Booker C. Bense</span></div>
</div>
</div>
</div>
</div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-kd-a" style="border: 0px; margin: 0px 0px 0px 44px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
</div>
</div>
<div class="H1SYRSB-wb-b" style="border: 0px; margin-bottom: 0px; margin-left: 39px; margin-right: 0px !important; margin-top: 0px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<br /></div>
</div>
</div>
</div>
</div>
<div class="H1SYRSB-wb-W H1SYRSB-vb-n H1SYRSB-wb-Y" style="border-bottom-color: transparent; border-bottom-style: solid; border-left-color: transparent; border-left-style: solid; border-top-color: rgb(221, 221, 221); border-top-style: solid; border-width: 1px 0px 1px 1px; margin: 0px; padding: 4px 0px; position: relative; vertical-align: baseline;" tabindex="0">
<input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; color: #222222; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><div class="H1SYRSB-wb-W" style="border-bottom-color: transparent; border-bottom-style: solid; border-left-color: transparent; border-left-style: solid; border-top-color: transparent; border-top-style: solid; border-width: 1px 0px 1px 1px; margin: 0px; padding: 0px; position: relative; vertical-align: baseline;">
<div class="H1SYRSB-wb-F" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-wb-x" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;" tabindex="0">
<input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; color: #222222; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<table cellpadding="0" cellspacing="0" class="H1SYRSB-wb-O" style="border-spacing: 0px; border: 0px; margin: 0px; padding: 0px; vertical-align: baseline; width: 1120px;"><tbody style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<tr style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"><td class="H1SYRSB-wb-p" style="border: 0px; margin: 0px; padding: 5px; vertical-align: top; width: 34px;" valign="top"><div aria-label="José Valim's profile photo" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<img class="gwt-Image" height="34" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACIAAAAiCAYAAAA6RwvCAAAAnUlEQVR42u3UsQ2AIBAFUOemoKGESajYhA3oWABCfeabUKiFNijRX3xDzkt4ILLknGWGLHi01l4NIYQQQsirkFKKhBDEGLMFY9Qeh2BipdQuqD0OwS4cIaj9FzLNp5nmsPIe6am1SoxRvPfinBOt9RaMUcM79AyFpJTEWns6pMegB73DIFj1FaIHvcMgdxE934fw9yWEEEIIuQuZISuz4exV/fiHnwAAAABJRU5ErkJggg==" style="border: 0px; height: 34px; margin: 0px; padding: 0px; vertical-align: baseline; width: 34px;" width="34" /></div>
</td><td align="left" class="H1SYRSB-wb-R" style="border: 0px; cursor: pointer; height: 1.5em; margin: 0px; overflow: hidden; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; padding-top: 6px !important; vertical-align: top; white-space: nowrap;" valign="top"><span style="color: yellow;"><span class="_username" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"><span class="H1SYRSB-P-a" style="border: 0px; cursor: default; font-weight: bold; margin: 0px; padding: 0px; vertical-align: top;">José Valim</span></span> <span style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"></span></span></td><td align="right" style="border: 0px; margin: 0px; padding: 0px; vertical-align: top;" valign="top"><div class="H1SYRSB-wb-e" style="border: 0px; float: right; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 5px !important; padding: 0px; vertical-align: baseline; white-space: nowrap; zoom: 1;">
<span style="color: yellow;"><div aria-disabled="false" aria-label="Post reply" class="H1SYRSB-l-a jfk-button-standard H1SYRSB-l-f H1SYRSB-wb-i H1SYRSB-wb-A" role="button" style="-webkit-box-shadow: none; background-color: whitesmoke; background-image: -webkit-linear-gradient(top, rgb(245, 245, 245), rgb(241, 241, 241)); border-bottom-left-radius: 2px; border-bottom-right-radius: 0px; border-top-left-radius: 2px; border-top-right-radius: 0px; border: 1px solid rgba(0, 0, 0, 0.0980392); box-shadow: none; cursor: pointer; display: inline-block; font-size: 11px; font-weight: bold; height: 27px; line-height: 27px; margin: 0px; min-width: 34px; outline: 0px; padding: 0px 8px; text-align: center; vertical-align: baseline;" tabindex="0">
<img aria-hidden="true" class="gwt-Image H1SYRSB-l-d" height="21" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABUAAAAVCAYAAACpF6WWAAAAdklEQVR42mNgGAVDHswnwCfLwP94+CSDBqgBCWh8sg1NgGruR+PD8HogLgBiBVINXI/DQHQ8nxjDYYoV0Pj48HsgNqCmS0k2GFeY7ofi+2ji94kNW1js9+OIfQGomvto6qmSTgWgQXae2jlKgJggIDeNjwIaAQBDo0wVdvRN2QAAAABJRU5ErkJggg==" style="border: 0px; margin: -2px 0px 0px; opacity: 0.667; padding: 0px; vertical-align: middle;" width="21" /><input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><span class="H1SYRSB-l-e" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"></span></div>
<span class="H1SYRSB-wb-i H1SYRSB-wb-f" style="border: 0px; display: inline-block; height: 27px; margin: 0px; padding: 0px; text-align: center; vertical-align: baseline;"><div aria-controls="gwt-uid-11844" aria-haspopup="true" aria-label="More message actions" class="H1SYRSB-l-a jfk-button-standard H1SYRSB-n-c H1SYRSB-wb-k H1SYRSB-l-i H1SYRSB-l-h" id="b_action_w_7Y6ZTqPvgJ" role="button" style="-webkit-box-shadow: none; background-color: whitesmoke; background-image: -webkit-linear-gradient(top, rgb(245, 245, 245), rgb(241, 241, 241)); border-bottom-left-radius: 0px; border-bottom-right-radius: 2px; border-top-left-radius: 0px; border-top-right-radius: 2px; border: 1px solid rgba(0, 0, 0, 0.0980392); box-shadow: none; cursor: pointer; display: inline-block; font-size: 11px; font-weight: bold; height: 27px; line-height: 27px; margin: 0px 0px 0px -1px; min-width: 0px; outline: 0px; padding: 0px 4px; vertical-align: baseline; width: 13px;" tabindex="0">
<input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><span class="H1SYRSB-l-e" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"><span style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"></span></span><span class="H1SYRSB-n-a" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"><span class="H1SYRSB-n-b" role="menu" style="background-image: url(data:image/png; background-position: 0px 0px; background-repeat: no-repeat no-repeat; border: 0px; display: inline-block; height: 4px; margin: 0px 0px 1px; opacity: 0.8; overflow: hidden; padding: 0px; vertical-align: baseline; width: 7px;"></span></span></div>
</span></span></div>
<div class="H1SYRSB-wb-R H1SYRSB-wb-T" style="border: 0px; cursor: pointer; height: 1.5em; margin: 0px; overflow: hidden; padding-bottom: 5px; padding-left: 5px; padding-right: 5px; padding-top: 6px !important; vertical-align: baseline; white-space: nowrap;">
<span class="H1SYRSB-wb-Q H1SYRSB-b-Qb" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;" title="Monday, February 23, 2015 at 10:22:41 AM UTC-8"><span style="color: yellow;">Feb 23</span></span></div>
</td></tr>
</tbody></table>
</div>
</div>
<div class="H1SYRSB-wb-gb" style="border: 0px; float: left; font-family: Arial, Helvetica, sans-serif; font-size: 13px; margin: 0px 10px; padding: 0px; position: relative; top: -4px; vertical-align: baseline;">
<div aria-label="Star this post" aria-pressed="false" class="H1SYRSB-K-d" role="button" style="border: 0px; cursor: pointer; display: inline-block; height: 13px; margin: 0px; padding: 5px; text-align: center; vertical-align: baseline; width: 13px;" tabindex="0">
<span style="color: yellow;"><input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><div class="H1SYRSB-K-c" style="background-image: url(https://ssl.gstatic.com/ui/v1/star/star2.png); background-position: initial initial; background-repeat: no-repeat no-repeat; border: 0px; height: 13px; margin: 0px; padding: 0px; vertical-align: baseline; width: 13px;">
</div>
<span style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;"></span></span></div>
</div>
<div class="H1SYRSB-kd-a" style="border: 0px; font-size: 13px; margin: 0px 0px 0px 44px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<div class="H1SYRSB-wb-P" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;" tabindex="0">
<input role="presentation" style="border-bottom-left-radius: 1px; border-bottom-right-radius: 1px; border-color: rgb(192, 192, 192) rgb(217, 217, 217) rgb(217, 217, 217); border-style: solid; border-top-left-radius: 1px; border-top-right-radius: 1px; border-width: 1px; font-family: Arial, Helvetica, sans-serif; height: 1px; line-height: 27px; opacity: 0; overflow: hidden; padding-left: 8px; position: absolute; vertical-align: top; width: 1px; z-index: -1;" tabindex="-1" type="text" /><div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; margin: 0px; overflow: auto; padding: 0px; vertical-align: baseline;">
<div dir="ltr" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="gmail_quote" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-Db-b" style="border: 0px; font-family: Arial, Helvetica, sans-serif; margin: 0px; padding: 0px; vertical-align: baseline;">
<blockquote class="gmail_quote" style="border-left-color: rgb(204, 204, 204); border-left-style: solid; border-left-width: 1px; margin: 0px 0px 0px 0.8ex; padding-left: 1ex;">
<div dir="ltr" style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Should any examples in the documentation exactly match the results from running the code in iex?</span></div>
</div>
</blockquote>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;"><br /></span></div>
</div>
<div style="border: 0px; font-family: Arial, Helvetica, sans-serif; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">Yes. But you can always "cheat"!</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;"><br /></span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;"># Get the doc for the first function</span></div>
<div style="border: 0px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> [fun|_] = Code.get_docs(Atom, :docs) |> Enum.sort()</span></div>
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;"># Each clause is in the format</span></div>
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> {{_function, _arity}, _line, _kind, _signature, text} = fun</span></div>
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;"># Let's get the first line of the text</span></div>
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;">iex> String.split(text, "\n") |> Enum.at!(0)</span></div>
<div style="border: 0px; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: orange; font-family: Courier New, Courier, monospace;">"NOW YOU HAVE A STRING"</span></div>
<div style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
</div>
<div style="border: 0px; font-family: Arial, Helvetica, sans-serif; font-size: 12.8000001907349px; margin: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: yellow;">So you are showing the whole process without printing it all the way</span><span style="color: #222222;">.</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; margin: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; margin: 0px; padding: 0px; vertical-align: baseline;">
</div>
</div>
<div style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13px; margin: 0px; padding: 0px; vertical-align: baseline;">
<div class="H1SYRSB-kd-a" style="border: 0px; margin: 0px 0px 0px 44px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
</div>
</div>
<div class="H1SYRSB-wb-b" style="border: 0px; font-family: Arial, Helvetica, sans-serif; margin-bottom: 0px; margin-left: 39px; margin-right: 0px !important; margin-top: 0px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<div class="H1SYRSB-wb-a H1SYRSB-wb-cb" style="border: 0px; color: #222222; font-size: 13px; margin: 0px; padding: 4px 0px; position: relative; vertical-align: baseline;">
</div>
<div class="H1SYRSB-wb-b" style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13px; margin-bottom: 0px; margin-left: 39px; margin-right: 0px !important; margin-top: 0px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<br /></div>
<div class="H1SYRSB-wb-b" style="border: 0px; color: #222222; font-family: Arial, Helvetica, sans-serif; font-size: 13px; margin-bottom: 0px; margin-left: 39px; margin-right: 0px !important; margin-top: 0px; padding: 0px 0px 0px 5px; vertical-align: baseline;">
<br /></div>
<span style="color: white;">This is a really nice way to take very complex dense output and recast it in a simple example that makes the return value of the function much clearer. </span></div>
</div>
</div>
</div>
</div>
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-6144319221316603092015-01-08T14:19:00.001-08:002015-05-13T11:09:14.289-07:00Using ex_doc to create a github web site for your elixir project. 1. Use mix docs to create docs dir.<br />
<br />
You'll need to add these to the deps section for your project in mix.exs<br />
<br />
[{:earmark, "~> 0.1", only: :dev},<br />
{:ex_doc, "~> 0.5", only: :dev}]<br />
<br />
Then run<br />
<br />
mix deps.get<br />
mix deps.compile<br />
mix docs<br />
<br />
This will create a doc subdir with an html documention for your project very similar to<br />
that used by the standard elixir docs.<br />
<br />
What you need to do know is to install those html pages into the special git branch that<br />
allows you to create a user.github.io/project website.<br />
<br />
2. Follow manual gh_pages instructions on github.<br />
<br />
https://help.github.com/articles/creating-project-pages-manually/<br />
<br />
git clone git@github.com:user/project.git project-gh-pages<br />
<br />
cd project-gh-pages<br />
<br />
git checkout --orphan gh-pages<br />
<br />
cp -r ../project/doc/* .<br />
git add --all<br />
<br />
( Note this is about the only time I would ever use git add --all )<br />
<br />
git commit -a -m "Ex doc output"<br />
git push origin gh-pages<br />
<br />
After a few minutes the web pages should show at the url<br />
<br />
http://user.github.io/project<br />
<br />Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com1tag:blogger.com,1999:blog-8030977544482607406.post-24762387678289737062014-11-14T08:40:00.000-08:002015-05-19T10:37:35.642-07:00Direct link to host in Nagios via Apache RewriteOne of the people using our nagios server asked the other day:<br />
<br />
"I know this doesn't work, but can https://nagios.our.site/ourhost go directly to the page for host?"<br />
<br />
I explained that the frames inside the current version of Nagios that we use were simply<br />
links to cgi urls like<br />
<br />
https://nagios.our.site/cgi-bin/status.cgi?type=1&host=ourhost<br />
<br />
and that you could type that url into the browser and get a direct link. As I was explaining this I realized that it would not be too hard to use Apache Rewrite to create a url like this<br />
<br />
https://nagios.our.site/host/ourhost<br />
<br />
This is what I added to the nagios.conf file in /etc/httpd/conf.d to accomplish this.<br />
<span style="background-color: #444444; color: yellow; font-size: x-small;"><br /></span>
<span style="background-color: #444444;"><span style="font-size: x-small;"><b><span style="color: yellow;"> <span style="font-family: 'Courier New', Courier, monospace;">R</span></span></b><b><span style="color: yellow;"><span style="font-family: 'Courier New', Courier, monospace;">ewriteEngine On</span></span></b></span></span><br />
<span style="color: yellow; font-family: 'Courier New', Courier, monospace; font-size: x-small;"><b style="background-color: #444444;"> RewriteLog /etc/httpd/logs/ssl_rewrite_log</b></span><br />
<span style="background-color: #444444; color: yellow; font-size: x-small;"><br />
<span style="font-family: Courier New, Courier, monospace;"><b> # Rewrite host/foo to basic cgi view url</b></span></span><br />
<span style="font-size: x-small;"><span style="color: yellow; font-family: Courier New, Courier, monospace;"><b style="background-color: #444444;"> RewriteRule ^/host/(.*\.cgi)$ /cgi-bin/$1 </b></span><b style="background-color: #444444; color: yellow; font-family: 'Courier New', Courier, monospace;">[L,PT]</b><b style="background-color: #444444; color: yellow; font-family: 'Courier New', Courier, monospace;"> </b></span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><b style="background-color: #444444;"> RewriteRule ^/host/([a-z0-9\-]+*)$ /cgi-bin/status.cgi?host=$1 [L,PT]</b></span><br />
<span style="color: yellow; font-family: Courier New, Courier, monospace; font-size: x-small;"><b style="background-color: #444444;"> </b></span><br />
<span style="background-color: #444444;"><br /></span>
The first rule rewrites any links clicked from with the initial page to correctly redirect to the appropriate CGI, the second rewrites the /host/ourhost to be a CGI url pointing to the basic status page. The options are important, the first L means "last rule, stop attempting to rewrite the url", PT means that it is not a plain file, but needs to go back through the URI engine to be resolved to a cgi.<br />
<br />
It's a simple hack, but people seem to like it. Which is probably more a statement about the Nagios dashboard than the utility of this tweak.Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-58403200202891354192014-09-15T10:52:00.002-07:002014-09-15T10:52:18.625-07:00Elixir command line parsingI haven't had much luck finding a complete example of command line parsing in Elixir, so I thought I'd share what I've come up with so far.<br />
<br />
This example shows uses a Map to ultimately store the options as key, value pairs. This implies that there can be only one value for any given option.<br />
<br />
<pre><code><span style="color: cyan;">
def main(args) do
args |> parse_args |> process
end
</span></code></pre>
<br />
This function defines the main flow of control in the program.
<br />
<pre><code><span style="color: cyan;">
def parse_args(args) do
options = %{ :count => @max_ofiles ,
:type => @default_type
}
cmd_opts = OptionParser.parse(args,
switches: [help: :boolean , count: :integer],
aliases: [h: :help, c: :count])
case cmd_opts do
{ [ help: true], _, _} -> :help
{ [], args, [] } -> { options, args }
{ opts, args, [] } -> { Enum.into(opts,options), args }
{ opts, args, bad_opts} -> {
Enum.into(merge_opts(opts,bad_opts),
options),
args
}
_ -> :help
end
end
</span></code></pre>
<br />
The main command line argument parsing function. It sets up an option map
with default values and then merges in any values from the command line.
It also allows undefined options, see <code>rehabilitate_args</code> below.
<br />
<br />
<pre><code><span style="color: cyan;">
def merge_opts(opts,bad_opts) do
bad_opts |> rehabilitate_args |> Keyword.merge(opts)
end
</span></code></pre>
<br />
<br />
A simple helper function to make the main parse routine less complicated.
<br />
<br />
<pre><code><span style="color: cyan;">
def rehabilitate_args(bad_args) do
bad_args
|>
Enum.flat_map(fn(x) -> Tuple.to_list(x) end)
|>
Enum.filter_map(fn(str) -> str end,
fn(str) ->
String.replace(str, ~r/^\-([^-]+)/, "--\\1")
end )
|>
OptionParser.parse
|>
Tuple.to_list
|>
List.first
end
</span></code></pre>
<br />
The function <code> rehabilitate_args </code> is something I've included since my application will use plugins and I would like plugin authors to be able to simply use command line args w/o a complicated interface. This might or might not be a good idea and is mostly included as an example of how to handle bad_args if you want to. If you use <code>:strict</code> rather than <code>:switches</code> in the <code>OptionParser.parse</code> call undefined options will automatically generate an error.
<br />
<pre><code><span style="color: cyan;">
def process(:help) do
IO.puts @module_doc
System.halt(0)
end
def process({options,args}) do
AllTheRealWork.start
end
</span></code> </pre>
<br />
This is an example of using elixir's pattern matching in function definitions to allow it to route flow of control in the program. The value returned from the pargs_args call will determine which version of the process function gets run. This code in actual use can be found in my github repo <a href="https://github.com/bbense/elixgrep">https://github.com/bbense/elixgrep</a>Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-46338930914700508592014-09-12T18:34:00.000-07:002014-09-12T18:34:33.683-07:00Elixir spawn/1 and spawn/3 Since I spent an afternoon banging my head against this, I thought I'd blog it.<br />
<br />
If you have elixir code that dies with this message<br />
<br />
<span style="background-color: red;"><br /></span>
<span style="background-color: red;"><code>
17:39:25.661 [error] Error in process <0.60.0> with exit value: {undef,[{'Elixir.MyFunModule',fun_stuff,[],[]}]}<br />
</code>
</span><br />
<div>
<br /></div>
when you do this<br />
<br />
<code style="background-color: blue;">pid = spawn( MyFunModule, :fun_stuff, [] )</code><br />
<br />
But runs just fine when you do this<br />
<br />
<code> <span style="background-color: blue;">pid = spawn( fn -> MyFunModule.funstuff([]) end )</span></code><br />
<br />
The problem is the arity of the function you are using in the spawn/3 version.<br />
Elixir uses the count of the elements in the third argument to spawn/3 to<br />
find the function to spawn in the subprocess. In the first version it is looking<br />
for a <code style="background-color: blue;">MyFunModule.fun_stuff/0</code> which does not exist.
If you change<br />
<br />
<span style="background-color: blue;">
</span><code><span style="background-color: blue;">pid = spawn( MyFunModule, :fun_stuff, [] )</span>
</code><br />
to<br />
<br />
<span style="background-color: blue;">
</span><code><span style="background-color: blue;">pid = spawn( MyFunModule, :fun_stuff, [[]] )</span>
</code><br />
it will then run without the process error above.
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-78450512056496573322014-08-21T23:14:00.001-07:002014-08-27T10:33:49.262-07:00Multi-Core, Languages and the Dreaded GIL For a very long time[1], I have always thought that the only way to deal with the coming multi-core explosion is to move to a language that deals with concurrency as a core feature in the language. This is hardly a novel idea. It has always been obvious that threaded code in standard languages is next to impossible to write correctly. This implies that the only way to deal with threads is to hide them in an abstraction in which you can devote enough resources to ensure a reasonably robust implementation. Writing thread safe code is extremely difficult and requires a huge payback to justify the effort. It's exactly the same reasoning behind allowing the kernel to do memory and process management, rather than having every application write it's own OS.<br />
<br />
In the face of this I have made a serious effort to learn Erlang, Go and dip my toe in functional languages in an attempt to be ready for adapting to this multi-core world. As much as I love Ruby,<br />
it was obvious that it simply would not be a reasonable solution for a machine with 32 cores and above. It and many of the other "scripting" languages, depend on having a lock on the dreaded GIL[2]. This is generally not a problem for code that runs websites due to the fact that they are largely I/O bound[3]. However, for any other workload that requires CPU these languages would be relegated to the "fancy shell script" problem domain. You simply wouldn't write anything serious in these languages, since they can't effectively use 80-90% of the available resources on a high end server.<br />
<br />
Recently, I have begun to rethink this assumption. The reality is that most 32 core machines are running VM's in one form or another. Docker[4], in particular, provides a highly efficient means of simulating a single core OS for a single core process. A common theme of the "multi-core" ready languages is to avoid the problems of threading by switching the model from shared memory to very fast messaging. If it is possible to create an interprocess message system between Docker images that can compete with speed of Go channels or Erlang messaging, then it should be possible for languages like Ruby to be a reasonable alternative in this brave new world.<br />
<br />
There was an attempt to allow this kind of distributed processing in Ruby, DRb. It never really gained wide usage primarily due to the latency and AAA[5] issues. However, if there was an API that allowed communication between Docker instances that could be both fast and trusted, it would be possible to revive Drb and similar "remote object" implementations. I think this is the only workable way forward for the "interpreted" languages. Essentially you have to convert the message sending primitive to sending a message to remote[6] objects. This isn't as hard as it looks at first blush, Docker images are simply processes running in the same kernel on the same machine. They just have different ideas about their various interfaces to the rest of the world. It should be possible to "short-circuit" these interfaces to allow them to communicate at something close to the speeds available to more concurrency aware languages.<br />
<br />
[ After a couple days of thinking about what this really means.]<br />
<br />
This idea looks more like "sprinkle current fairy dust" on a hard problem, than an actual solution. I think what I was trying to wrap my brain around was the idea of implementing something like the old sun RPC interface via standard ports as a way to allow you to use Docker to avoid all the overhead of<br />
doing message passing between ruby processes via pipes, etc...<br />
<br />
I think there is something there, but it's far from fleshed out enough. Maybe just creating a lightweight micro services API and using Docker to allow those API's to talk on the same machine<br />
might be enough. I am well aware of the first rule of Distributed Objects.<br />
<br />
http://martinfowler.com/articles/distributed-objects-microservices.html<br />
<br />
<br />
<br />
<br />
<br />
<br />
[1]- Since about 1997 or so (i.e. when I first attempted to write pthreaded code ).<br />
<br />
[2]- Global Interpreter Lock https://en.wikipedia.org/wiki/Global_Interpreter_Lock<br />
<br />
[3]- i.e. Turning database entries into web pages and web forms into database entries.<br />
<br />
[4] - https://www.docker.com/<br />
<br />
[5]- Authentication, Authorization and Accounting<br />
<br />
[6]- If they are in the same memory accessed by the same cpu, they really aren't "remote"Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-53536800101471124942013-11-08T16:52:00.000-08:002013-11-08T16:52:24.647-08:00If you really think your employee's performance is on a bell curve, you should fire your HR department. A recent kerfuffle over in Yahoo Land has got me thinking about the mis-application of statistics.<br />
<br />
<a href="http://allthingsd.com/20131108/because-marissa-said-so-yahoos-bristle-at-mayers-new-qpr-ranking-system-and-silent-layoffs/">http://allthingsd.com/20131108/because-marissa-said-so-yahoos-bristle-at-mayers-new-qpr-ranking-system-and-silent-layoffs/</a><br />
<br />
The assumption is that employee performance follows a bell curve. But you only get a bell curve in your sample of a larger population if your sample is random and the larger population is also a bell curve.<br />
<br />
What this means is that if your employee's performance truly follows a bell curve, then you should fire your HR department and completely replace all those expensive people with a random process that just picks a resume out of the hat. Frankly, I suspect you'd be hard pressed to tell the difference in most organizations.Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-87403759946885405372013-10-10T23:05:00.000-07:002013-10-10T23:26:13.913-07:00Person Capital and the Government Shutdown<br />
One of the mantras of the information age is<br />
<br />
"People are Capital"<br />
<br />
<div style="text-align: justify;">
In other words, the experience and knowledge of your workforce is a significant capital investment. It is the only capital investment that can walk out the door whenever it get's pissed off enough and right now all the best people employed by either directly or indirectly by the federal government are polishing their resumes and checking just exactly who they know on LinkedIn. I know because that's exactly what I've been doing.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: left;">
The great lie that has been put out there is that government jobs are cushy and easy and only incompetent people would take them. Well, one way or another I've been employed by the Federal government indirectly for the every job I've ever had since I stopped waiting tables, cooking or washing dishes in 1987. The reality is that most people in the government work long hours for relatively low wages given their expertise and education. Despite every attempt to belittle it on Fox News, Public service is alive and well. Many of the people not getting paychecks are hard working competent people that</div>
<div style="text-align: left;">
could make more money in the private sector, but choose to work for the federal paycheck because they believe in public service. For me, it matters that what I do advances the knowledge of mankind in some way. </div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
I remember a conversion in high school (1978) about the future and I said my "dream job" would be to work at a "national lab doing physics research". One of my friends that had more exposure to the larger world said "Those are terrible jobs". </div>
<div style="text-align: left;">
<br /></div>
It's really odd that we were both right. My job is my dream job, but it is also terrible in that I never get the resources to do it right. The current drama in Washington is the last straw for many of us; the best people currently employed by the government are all looking around for stable paychecks.<br />
<br />Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-62046788205347976152013-09-16T11:46:00.000-07:002013-09-16T11:46:01.002-07:00Chef Encrypted Data Bags Are A Code Smell. <br />
<a href="http://docs.opscode.com/essentials_data_bags.html" target="_blank">Chef databags</a> are a centralized key/value store for json based data in the Chef environment. Encrypted databags are exactly what is advertised on the box, the data is encrypted with an external key before uploading to the data store.<br />
<br />
Encrypted databags are meant to help solve the problem of dealing with identifier information that you want automatically installed, but that you need to keep private. Examples are database login passwords, the private key of a public key pair and other tokens that allow a server access to additional services.<br />
<br />
Encrypted databags provide protection against two kinds of access:<br />
<br />
<b>Implicit Access</b>:<br />
<br />
This is defined to mean access outside the chef protocol to the underlying data store on your chef server. If you're using a service like Hosted Chef or just practicing general good data hygiene, DB access to the server should only expose "public" data if at all possible.<br />
<br />
<b>Explicit Access</b>:<br />
<br />
Explicit access is via the standard chef protocol were using the ssl keypair created as part of the chef client bootstrap to access data in the chef datastore or via a knife command using an admin key pair.<br />
<br />
The problem with encrypted databags in both use cases is that they only appear to solve the underlying problem by moving it out of the chef workflow. In order to use either protection effectively, you need to create an "out of band" system to manage the shared secret required to access the databag.<br />
<br />
In the implicit access protection case, this is likely a worthwhile and manageable cost since the key only needs to be secret from the chef server. However, this only protects against read-only implicit access. If the bad guy has write implicit access to your chef datastore, the game is over on your next client chef run.<br />
<br />
The explicit access case is where the real problems arise. In this case, the intention is to prevent some admins/hosts from access to private data. This however creates an unfortunate side effect in that access to the shared secret becomes an "invisible access control list". Using an encryption key as an authorization object creates problems since you destroy the chain of identity. All you ever know is that "somebody with the key" accessed the data. You need to create a separate tracking and access control system to provide an audit-able trail. Encrypted databags don't solve a problem in this case, they just create a whole new set of problems.<br />
<br />
<a href="https://github.com/Nordstrom/chef-vault" target="_blank">Chef Vault</a> is an attempt to get around this problem by creating access control lists using the available private key on each chef client. Without the strong ACL system that either Private or Hosted Chef provide, this is probably the simplest workable solution. Any other solution should implement all of the features of Chef Vault. The most general solution in the explicit access case is ACL's based on the ssl identity of the client. There are many other objects in Chef on which the ACL system of the Opscode Chef would provide useful security boundaries.<br />
<br />
It's important to remember that more keys is not better security. Encryption keys should be used to provide data integrity, privacy and authentication. (i.e. they answer the who questions, who are you? who am I? who sent that message? ). They should never be used to answer the what questions. ( what can I read? what can I write?, etc ).<br />
<br />
Providing secrets in a scalable and secure fashion is still a largely unsolved problem. But any solution that attempts to use only shared secret encryption will not scale. Public key encryption and rings of shared access seem to be the only workable way forward and every chef client and admin has key pair already available. Any scalable solution should be using this existing identity. If you are using encrypted databags to control explicit access, then you are building in scaling, access and audit problems for the future.<br />
<br />
<br />Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-8420275735895237132013-05-23T21:56:00.000-07:002013-05-23T22:02:02.088-07:00There Are Only Two Ways to Save Money with SoftwareTo paraphrase a civil war general:<br />
<br />
<b> "The only good code is deleted code."</b><br />
<br />
I have recently been in several meetings where the phrase<br />
<i><br /></i>
<i> "We'll save resources by doing things with new software",</i><br />
<u style="font-style: italic;"><br /></u>
has been used a justification for change.<br />
<br />
This is one of the great lies that drives Silicon Valley. Every line of software is a cost liability.<br />
<br />
The only way to save money with software is to delete the number of lines of software you run or to use software to make the problem somebody else's problem.
Every advance in computer software engineering has followed these two principals.<br />
<br />
Compiled languages follow this principal by both
reducing the number of lines of code required to solve a problem and also by handing the problem of turning those lines of code into executable instructions to the people that write compilers.<br />
<br />
Operating systems follow the same principal; instead of having to manage
access to millions of blocks in both memory and disk, you write enough software to make it the user's problem.<br />
<br />
The only way to save resources with new software is to delete old software, or to make the things that the old software was responsible for somebody else's problem.<br />
<br />
New software can make doing new things cheaper, but that is a completely different problem. Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-44073564888933698302013-05-22T15:10:00.001-07:002013-05-22T15:10:50.305-07:00How to build ruby 1.9.3 with libyaml in a funky placeThe only solution I've found is to hack the location of libyaml library and include files<br />
into the extconf.rb file in ext/psych<br />
<br />
<br />
<code>
header_dir = '/afs/slac/package/ruby/@sys/include'<br />
library_dir = '/afs/slac/package/ruby/@sys/lib'<br />
<br />
dir_config 'libyaml', header_dir, library_dir<br />
<div>
</code>
<br /></div>
Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-6387302075581236802012-10-12T21:59:00.000-07:002012-10-12T21:59:43.784-07:00Why Monitoring Sucks<b>It doesn't matter what software you use, the fundamental model behind all current monitoring software is broken. </b><br />
<br />
The idea the originally prompted me to create this blog still holds true, I never published it. But the basic tenent of that message still holds true.<br />
<br />
"Push Considered Harmful"<br />
<br />
The original idea I had was in the context of grid computing in the early 2000's. Early versions of the LHCG were based on the idea that a central scheduler could monitor the entire state of the "grid" and<br />
assign jobs to the hosts that had both the data and the available cycles. This failed badly, and the concept of "pilot jobs" invaded LHCG. In essense, pilot jobs flip the paradigm on it's head and turn "push" architectures into "pull" ones. Instead of a node passive waiting for the central service to ask<br />
it for state, the central service should be waiting for nodes to notify it for state. <br />
<br />
There have been no large scale push architectures that have actually ever worked outside a single organization. This is an important lesson for monitoring.<br />
<br />
The reason "monitoring sucks" is that it is entirely based on a push model. All current monitoring software assumes that you have a model of your infrastructure that can be mapped onto existing<br />
hardware either virtual or real. This is completely backwards.<br />
<br />
There will be no improvement in monitoring software until this model is thrown out and a "pull" or<br />
peer to peer model is created. There is a reason Nagios still exists, it's because everything else that<br />
has tried to replace it uses the same "push" model.<br />
<br />
What is needed is a subscribe model in which a node requests a monitoring service to "pull" service requests with a finite time limit. Similarly there needs to be a subscribe service for alerts. Monitoring configuration needs to be a dynamic service; any software that reads a static file for configuration is<br />
doomed to suck. <br />
<br />
The other fundamental problem in monitoring is the poor separation between alert and diagnostic services. For diagnostics, you want to monitor "everything", for alerts you should only be monitoring<br />
things that you can actually control. If it sends an alert, there needs to be procedure to handle the alert. <br />
<br />
Until monitoring services look a lot more like DHCP than DNS, they will continue to suck, regardless of the software used to implement them. Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0tag:blogger.com,1999:blog-8030977544482607406.post-70415117909947698042011-05-20T23:13:00.000-07:002011-05-20T23:28:06.554-07:00On the Eve of the FakapocalypseWhile scrolling through the channel guide I came across this gem. <div><br /></div><div><a href="http://en.wikipedia.org/wiki/Fail-Safe_(1964_film)">FailSafe </a></div><div><br /></div><div>And it reminded me that I was born into a world that really was 30 seconds from the apocalypse. We can laugh now at the fakapocalypse of the bad math of deluded visionaries, but not that long ago the world really was at the brink of total destruction. </div><div>We still have the capability to turn the world into an ash cinder, but we seem to have run out of enemies willing to put that much on the table. </div><div><br /></div><div>Maybe we still are doomed as a species, after all we have not escaped the brutal equations of Malthus. Finite resources and exponential population growth have only one outcome. I find some hope in the fact that more than cock roaches and deep sea tube worms might survive our extinction. </div><div><br /></div><div>But still, no matter how bleak our world view gets, it seems like there will be a world to view. That's a lot more than most people thought possible when I was born. </div><div><br /></div><div>- Booker C. Bense </div>Booker Bensehttp://www.blogger.com/profile/16889248159480666096noreply@blogger.com0