Thursday, July 16, 2015

Tilting at Windmills: Accessing Erlang man pages from Iex, Part 2:

Digging into the source code of iex, you find that the helpers are macros that pull apart the code and do introspection on the Elixir module to find the documentation. To get the documentation for a specific function in a module the code that eventually gets run is this:

iex> Code.get_docs(Atom, :docs)

[{{:to_char_list, 1}, 22, :def, [{:atom, [], nil}],
  "Converts an atom to a char list.\n\nInlined by the compiler.\n"},
 {{:to_string, 1}, 12, :def, [{:atom, [], nil}],
  "Converts an atom to string.\n\nInlined by the compiler.\n"}]

Where module is a Atom. All module names are Atoms. Code.get_docs returns a list of tuples describing the attributes of each of the functions exported by the module.

 {{function, arity }, line, kind, signature, text}

The initial tuple is the function name and arity. line is the line of the source file on which the function. kind is an atom that describes whether the exported function is a macro or a function.
signature is a list of tuples that define the arguments that the function takes, and lastly text is
the markdown documentation for the function.

If you're willing to fudge things a bit you can duplicate the parts of this function that the iex h command uses by parsing erlang man pages. Every erlang module supports the function module_info.

iex> Atom.module_info(:exports)

[__info__: 1, to_string: 1, to_char_list: 1, module_info: 0, module_info: 1]

iex> :et.module_info(:exports) 

[trace_me: 4, trace_me: 5, phone_home: 4, phone_home: 5, report_event: 4,
 report_event: 5, module_info: 0, module_info: 1]

Since the erlang man pages are created by a standard xml -> nroff process, it's relatively straightforward to split the nroff man page into documentation sections and to identify the function
documented by that section. So creating an ErlangCode.get_docs is possible by mapping the text
from the man page to the appropriate tuple in the export list. You can find code that does this in
the Erlman project on github.

Erlman

Unfortunately, this is kind of a dead end since it's not general enough to be included in the main elixir source code. It requires that the erlang man pages be installed and that is simply something that can not be counted on. It's also unclear if the current version will even work on Windows with the man pages installed.

My next thought is "Is there a hook into iex commands that I can use to make this an optional addon?"

Friday, June 19, 2015

Tilting At Windmills, Accessing Erlang man pages from Iex Part: 1

In the process of learning how to get a cryptographic hash of a file in Elixir, I found myself switching back and forth from iex to erl -man to better understand how
to use the available Erlang functions in Elixir. After all there are over 500 modules in the standard Erlang libraries that are available in Elixir. Wouldn't it be nice I thought to just be able to type

iex> h :crypto.hash 

and get some basic information about the functions. Elixir already supports listing all the functions available in both Elixir and Erlang modules, using the module.TAB sequence

iex> h :crypto.TAB 
 
If we have Erlang installed, the man pages must be there somewhere. And of course one of the tenants of Iex is that any functionality should be available in the Windows version if at all possible. While creating a command to shell out to erl -man would be simple, it wouldn't be portable.

Erlang installs it's man pages in a separate path to avoid conflicts with the standard unix man pages but puts them in the standard man/man3 locations. The convention is that the man page for an Erlang module is the Erlang module name followed by section number. The standard Erlang functions are documented in the erlang.3 man page.

So my next step was to reimplement erl -man in purely Elixir code. The first part went really quickly, finding the man page for a given module was relatively straight forward. We just find where the erl executable is and work backwards from that to find where the erlang man pages are and then search for the module name.

 def manpath do
    start = System.find_executable("erl")
    case start do
      nil -> nil
      _   -> find_manpath(start) 
    end
  end

  defp find_manpath(erl_path) do
    mpath = erl_path |>
            Path.split |>
            Stream.scan(&Path.join(&2,&1)) |>
            Enum.filter( fn(p)  -> File.exists?(Path.join([p,"man","man3","ets.3"])) end ) |>
            List.last
    if mpath , do: Path.join(mpath,"man"), else: nil
  end


After that I implemented a very simple-minded man nroff macro to markdown translator and was able to display an Erlang man page from the iex prompt. During this process I noticed that the Erlang man pages use a very consistant, small subset of the man nroff macro package. This lead me to believe that it would be possible to extract the specific function documentation from the man pages.

It turns out that Erlang documentation is maintained in XML and translated by an xmerl based program into both man pages and html documentation. Unfortunately, this XML source is not included in the default binary distributions of Erlang. Unix versions get the man pages ( and html usually ), but the Windows distributions only get the html. Parsing the html is a lot more complicated that dealing with the nroff and there are no HTML parsers in the standard Elixir libs.

At this point, I decided to dig a bit deeper and figure out just how

iex> h Atom.to_char 

really works.

Friday, June 5, 2015

Micro Benchmarking in Elixir using Benchfella

When I finished the last post, I had complete intentions of writing a post about how to use the Ports module in Elixir to interaction with the system shell to do I/O redirection. However, I now understand why people complain about Erlang documentation. Ports in Elixir aren't documented other than by a reference to the underlying Erlang libraries and the documentation in Erlang for ports is very incomplete. ( For example, what are the default drivers available in erlang?)

Ports are gateways between the Erlang VM and external codes and processes. The Erlang VM is sensitive to code "hanging" in any fashion and Ports need drivers that are aware of how to interact with the Erlang VM safely. However I was unable to find any clear documentation of just what drivers are available and the examples I found were inconsistent.

UPDATE: 

The types and drivers are documented in the man page for erlang in the open_port section. 

erl -man erlang


I did however find the Porcelain Elixir module that is both well documented and very straightforward to use. Having taken this long to get back to this, I'd just as soon go on to the actual benchmarks.

Benchfella is a micro benchmarking framework that works much like ExUnit does for testing and
you can use the same Dave Thomas hack for creating many tests that iterate over a list of values.

defmodule Hash do
  use Benchfella

  @lengths [1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,16777216]
  
  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**24 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_24",@chunk) 
    end
  end 

  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**26 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_26",@chunk)
    end
  end 

  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**28 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_28",@chunk)
    end
  end 


  def hash_test(file,chunk) do
   File.stream!(file,[],chunk) 
   |> 
    Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end ) 
   |> 
    :crypto.hash_final 
   |> 
    Base.encode16 
  end 

end

Benchfella runs each test for as many times as possible in a given interval ( the default is one second ) and returns the average time per test over that interval. Data from each run is stored on the filesystem so you can do comparisons between runs.  The plot below shows the results from using a chunk size in powers of 2 from 2**10 to 2**24 to hash a file of size 2**24.





The results are similar for files of size 2**26 and 2**28. As you can see there is a significant advantage to using a large chunk size. ( With an odd bump at 2**23 ) This test
was done on a MacBook Pro with 16gig of memory and an SSD disk drive.

This shows that using large binaries in Elixir ( and Erlang ) is generally the fastest way to deal with large data sets. Of course you need to make the tradeoff between total available memory and the number of binaries you want to process at a time.

The other benchmark I tested was to compare using the "chunk" method of hashing the file with a chunk size larger than the file and simply reading the entire file into a string and
computing it's hash. The simple read method was consistently twice as fast as the single chunk method. 

So for my resulting application I choose to pick a chunk size that allows the code to process multiple files at a time and chooses a method for computing the hash based on the
size of the file.

Thursday, May 14, 2015

Benchmarking Elixir File Hashing Step 1. Creating test data

In my previous post I outlined how to get a cryptographic hash of a file using Elixir. For large files, you want to chunk the file rather than read the whole thing into memory, it would be good to determine if there is an optimal chunk size to use to for passing into the encryption functions.
 iex> File.stream!("./known_hosts.txt",[],2048)
|> Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end )
|> :crypto.hash_final |> Base.encode16 "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

The first step in doing this is to create test data files of a specific length. An obvious approach is to simply open /dev/random and read until you've got enough sample data. A simple shell example.
 head -c 1024 < /dev/random > test.data 

Translating that to Elixir looks like
  iex(1)> File.stream!("/dev/random",[],1024) |> Enum.take(1) ** (File.Error) could not stream /dev/random: illegal operation on a directory (elixir) lib/file/stream.ex:81: anonymous fn/2 in Enumerable.File.Stream.reduce/3 (elixir) lib/stream.ex:1012: anonymous fn/5 in Stream.resource/3 (elixir) lib/enum.ex:1740: Enum.take/2
That sure looks like a bug, /dev/random is not a directory, but a char special device file. While the error message is misleading, there is no actual bug. Erlang will not open files for reading that it considers dangerous to the overall scheduler. In this case, /dev/random is a character special device file and since these kinds of files usually block on I/O, Erlang errs on the side of caution and will refuse to open the file. There is an exception in the Erlang code for /dev/null since that is considered safe for the scheduler. This post goes into the details.

Reading Device Files in Erlang

There are several ways to get around this problem. The first solution that springs to mind is actually one of the more difficult ones to do in Elixir. In many languages there is a system call that you can use to execute shell commands. Elixir has System.cmd, but it is relatively limited. You can specify the command to execute and the argument list, but you cannot use shell based I/O redirection.

The most straightforward Elixir solution is to use the rand_bytes function from the Erlang crypto library.
  iex(1)> File.write("test.data",:crypto.rand_bytes(1024))


But that isn't much fun, and while it solves this problem it doesn't give us a tool for interacting with external programs. We'll look at more general solutions in the next post...

Wednesday, April 29, 2015

How to get a hash of a file in Exilir

As the next step in learning about elixir I wanted to add a file validator module to my elixgrep project. Of course this requires as a first step taking the cryptographic hash of a file. I found this handy blog entry, but it didn't answer the whole problem. For small files, you can just read in the whole file into a string and hash it.

iex> :crypto.hash(:sha256,File.read!("./known_hosts.txt")) |> Base.encode16       "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

But there reaches a point were the file size is large enough that loading the whole contents in memory isn't performant and in some cases not feasible. My next idea was to use File.stream!
iex> File.stream!("./known_hosts.txt") |>
Enum.reduce(:crypto.hash_init(:sha256),
fn(line, acc) -> :crypto.hash_update(acc,line) end ) |>
:crypto.hash_final |> Base.encode16
"97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

However there is still a problem with this in that it assumes the file has appropriate line endings. For a cryptographic hash it makes more sense to divide up the file into equal byte length chunks. File.stream!/3 has two default arguements, modes and lines_or_bytes, if you want to stream in by byte_length use this form.

 iex> File.stream!("./known_hosts.txt",[],2048) |> Enum.reduce(:crypto.hash_init(:sha256),
fn(line, acc) -> :crypto.hash_update(acc,line) end ) |> :crypto.hash_final |> Base.encode16 "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 


 Now the interesting question becomes is there an optimal byte size to use for this hashing? STAY TUNED...

Thursday, April 16, 2015

Elixir functions are not Functions

I was banging my head against a simple elixir test this morning. It's a test that's been in my code for months and that never worked for some reason. This is a boiled down example:


defmodule RaiseTest do
        def testraise do
                raise "This is an error"
        end
end

defmodule RaiseTestTest do
  use ExUnit.Case
  test "assert_raise works" do
    assert_raise(RuntimeError, RaiseTest.testraise )
  end
end

mix test

  1) test assert_raise works (RaiseTestTest)
     test/raise_test_test.exs:4
     ** (RuntimeError) This is an error
     stacktrace:
       (raise_test) lib/raise_test.ex:4: RaiseTest.testraise/0
       test/raise_test_test.exs:5


Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 tests, 1 failures
My error was finally pointed out to me this morning on the elixir list and opened my eyes to a consistent flaw in my thinking about elixir. The fix of course is to actually provide a function to assert_raise, what I was actually providing was an expression that could return anything. 

 test "assert_raise works" do
    assert_raise(RuntimeError, fn -> RaiseTest.testraise end ) 
end

The key misunderstanding in my head was that function when used in the elixir context actually maps to what I think of as a function reference from my years of C programming. Module functions return expressions which can be anything. The test failed because it was calling TestRaisetest to see if it returned a function it could use in the test. 
When an elixir function requires a "function" as an argument, it really means a closure that can be executed.

Friday, April 3, 2015

Remember JSON is always valid YAML

While I don't mind YAML for relatively simple data files, I find it to be very difficult to use when there are more than one or two levels of nesting. If you have a relatively complex config file, then I find JSON to be a much easier to reason about and make any necessary changes.

For current versions of YAML, JSON files are a subset of the YAML standard. Thus any compliant YAML parser will also parson JSON.

The place where I use this trick the most often is in .kitchen.yml files used by TestKitchen when writing Chef cookbooks.  I start out by taking an existing .kitchen.yml file and converting it to
.kitchen.json using this website http://codebeautify.org/yaml-to-json-xml-csv. I then copy the
.kitchen.json to .kitchen.yml.