Tilting At Windmills, Accessing Erlang man pages from Iex Part: 1

In the process of learning how to get a cryptographic hash of a file in Elixir, I found myself switching back and forth from iex to erl -man to better understand how
to use the available Erlang functions in Elixir. After all there are over 500 modules in the standard Erlang libraries that are available in Elixir. Wouldn't it be nice I thought to just be able to type

iex> h :crypto.hash 

and get some basic information about the functions. Elixir already supports listing all the functions available in both Elixir and Erlang modules, using the module.TAB sequence

iex> h :crypto.TAB 
 
If we have Erlang installed, the man pages must be there somewhere. And of course one of the tenants of Iex is that any functionality should be available in the Windows version if at all possible. While creating a command to shell out to erl -man would be simple, it wouldn't be portable.

Erlang installs it's man pages in a separate path to avoid conflicts with the standard unix man pages but puts them in the standard man/man3 locations. The convention is that the man page for an Erlang module is the Erlang module name followed by section number. The standard Erlang functions are documented in the erlang.3 man page.

So my next step was to reimplement erl -man in purely Elixir code. The first part went really quickly, finding the man page for a given module was relatively straight forward. We just find where the erl executable is and work backwards from that to find where the erlang man pages are and then search for the module name.

 def manpath do
    start = System.find_executable("erl")
    case start do
      nil -> nil
      _   -> find_manpath(start) 
    end
  end

  defp find_manpath(erl_path) do
    mpath = erl_path |>
            Path.split |>
            Stream.scan(&Path.join(&2,&1)) |>
            Enum.filter( fn(p)  -> File.exists?(Path.join([p,"man","man3","ets.3"])) end ) |>
            List.last
    if mpath , do: Path.join(mpath,"man"), else: nil
  end

After that I implemented a very simple-minded man nroff macro to markdown translator and was able to display an Erlang man page from the iex prompt. During this process I noticed that the Erlang man pages use a very consistant, small subset of the man nroff macro package. This lead me to believe that it would be possible to extract the specific function documentation from the man pages.

It turns out that Erlang documentation is maintained in XML and translated by an xmerl based program into both man pages and html documentation. Unfortunately, this XML source is not included in the default binary distributions of Erlang. Unix versions get the man pages ( and html usually ), but the Windows distributions only get the html. Parsing the html is a lot more complicated that dealing with the nroff and there are no HTML parsers in the standard Elixir libs.

At this point, I decided to dig a bit deeper and figure out just how

iex> h Atom.to_char 

really works.

Micro Benchmarking in Elixir using Benchfella

When I finished the last post, I had complete intentions of writing a post about how to use the Ports module in Elixir to interaction with the system shell to do I/O redirection. However, I now understand why people complain about Erlang documentation. Ports in Elixir aren't documented other than by a reference to the underlying Erlang libraries and the documentation in Erlang for ports is very incomplete. ( For example, what are the default drivers available in erlang?)

Ports are gateways between the Erlang VM and external codes and processes. The Erlang VM is sensitive to code "hanging" in any fashion and Ports need drivers that are aware of how to interact with the Erlang VM safely. However I was unable to find any clear documentation of just what drivers are available and the examples I found were inconsistent.

UPDATE: 

The types and drivers are documented in the man page for erlang in the open_port section. 

erl -man erlang

I did however find the Porcelain Elixir module that is both well documented and very straightforward to use. Having taken this long to get back to this, I'd just as soon go on to the actual benchmarks.

Benchfella is a micro benchmarking framework that works much like ExUnit does for testing and
you can use the same Dave Thomas hack for creating many tests that iterate over a list of values.

defmodule Hash do
  use Benchfella

  @lengths [1024,2048,4096,8192,16384,32768,65536,131072,262144,524288,1048576,2097152,4194304,8388608,16777216]
  
  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**24 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_24",@chunk) 
    end
  end 

  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**26 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_26",@chunk)
    end
  end 

  for chunk <- @lengths do 
    @chunk chunk 
    bench "Hash 2**28 file by #{Integer.to_string(@chunk)}" do
      hash_test("./bench/data_2_28",@chunk)
    end
  end 


  def hash_test(file,chunk) do
   File.stream!(file,[],chunk) 
   |> 
    Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end ) 
   |> 
    :crypto.hash_final 
   |> 
    Base.encode16 
  end 

end

Benchfella runs each test for as many times as possible in a given interval ( the default is one second ) and returns the average time per test over that interval. Data from each run is stored on the filesystem so you can do comparisons between runs.  The plot below shows the results from using a chunk size in powers of 2 from 2**10 to 2**24 to hash a file of size 2**24.

The results are similar for files of size 2**26 and 2**28. As you can see there is a significant advantage to using a large chunk size. ( With an odd bump at 2**23 ) This test
was done on a MacBook Pro with 16gig of memory and an SSD disk drive.

This shows that using large binaries in Elixir ( and Erlang ) is generally the fastest way to deal with large data sets. Of course you need to make the tradeoff between total available memory and the number of binaries you want to process at a time.

The other benchmark I tested was to compare using the "chunk" method of hashing the file with a chunk size larger than the file and simply reading the entire file into a string and
computing it's hash. The simple read method was consistently twice as fast as the single chunk method. 

So for my resulting application I choose to pick a chunk size that allows the code to process multiple files at a time and chooses a method for computing the hash based on the
size of the file.

Benchmarking Elixir File Hashing Step 1. Creating test data

In my previous post I outlined how to get a cryptographic hash of a file using Elixir. For large files, you want to chunk the file rather than read the whole thing into memory, it would be good to determine if there is an optimal chunk size to use to for passing into the encryption functions.

 iex> File.stream!("./known_hosts.txt",[],2048)
|> Enum.reduce(:crypto.hash_init(:sha256),fn(line, acc) -> :crypto.hash_update(acc,line) end )
|> :crypto.hash_final |> Base.encode16 "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

The first step in doing this is to create test data files of a specific length. An obvious approach is to simply open /dev/random and read until you've got enough sample data. A simple shell example.

 head -c 1024 < /dev/random > test.data 

Translating that to Elixir looks like

  iex(1)> File.stream!("/dev/random",[],1024) |> Enum.take(1) ** (File.Error) could not stream /dev/random: illegal operation on a directory (elixir) lib/file/stream.ex:81: anonymous fn/2 in Enumerable.File.Stream.reduce/3 (elixir) lib/stream.ex:1012: anonymous fn/5 in Stream.resource/3 (elixir) lib/enum.ex:1740: Enum.take/2

That sure looks like a bug, /dev/random is not a directory, but a char special device file. While the error message is misleading, there is no actual bug. Erlang will not open files for reading that it considers dangerous to the overall scheduler. In this case, /dev/random is a character special device file and since these kinds of files usually block on I/O, Erlang errs on the side of caution and will refuse to open the file. There is an exception in the Erlang code for /dev/null since that is considered safe for the scheduler. This post goes into the details.

Reading Device Files in Erlang

There are several ways to get around this problem. The first solution that springs to mind is actually one of the more difficult ones to do in Elixir. In many languages there is a system call that you can use to execute shell commands. Elixir has System.cmd, but it is relatively limited. You can specify the command to execute and the argument list, but you cannot use shell based I/O redirection.

The most straightforward Elixir solution is to use the rand_bytes function from the Erlang crypto library.

  iex(1)> File.write("test.data",:crypto.rand_bytes(1024))

But that isn't much fun, and while it solves this problem it doesn't give us a tool for interacting with external programs. We'll look at more general solutions in the next post...

How to get a hash of a file in Exilir

As the next step in learning about elixir I wanted to add a file validator module to my elixgrep project. Of course this requires as a first step taking the cryptographic hash of a file. I found this handy blog entry, but it didn't answer the whole problem. For small files, you can just read in the whole file into a string and hash it.

iex> :crypto.hash(:sha256,File.read!("./known_hosts.txt")) |> Base.encode16       "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

But there reaches a point were the file size is large enough that loading the whole contents in memory isn't performant and in some cases not feasible. My next idea was to use File.stream!

iex> File.stream!("./known_hosts.txt") |>
Enum.reduce(:crypto.hash_init(:sha256),
fn(line, acc) -> :crypto.hash_update(acc,line) end ) |>
:crypto.hash_final |> Base.encode16
"97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

However there is still a problem with this in that it assumes the file has appropriate line endings. For a cryptographic hash it makes more sense to divide up the file into equal byte length chunks. File.stream!/3 has two default arguements, modes and lines_or_bytes, if you want to stream in by byte_length use this form.

 iex> File.stream!("./known_hosts.txt",[],2048) |> Enum.reduce(:crypto.hash_init(:sha256),
fn(line, acc) -> :crypto.hash_update(acc,line) end ) |> :crypto.hash_final |> Base.encode16 "97368E46417DF00CB833C73457D2BE0509C9A404B255D4C70BBDC792D248B4A2" 

 Now the interesting question becomes is there an optimal byte size to use for this hashing? STAY TUNED...

Elixir functions are not Functions

I was banging my head against a simple elixir test this morning. It's a test that's been in my code for months and that never worked for some reason. This is a boiled down example:

defmodule RaiseTest do
        def testraise do
                raise "This is an error"
        end
end

defmodule RaiseTestTest do
  use ExUnit.Case
  test "assert_raise works" do
    assert_raise(RuntimeError, RaiseTest.testraise )
  end
end

mix test

  1) test assert_raise works (RaiseTestTest)
     test/raise_test_test.exs:4
     ** (RuntimeError) This is an error
     stacktrace:
       (raise_test) lib/raise_test.ex:4: RaiseTest.testraise/0
       test/raise_test_test.exs:5


Finished in 0.03 seconds (0.03s on load, 0.00s on tests)
1 tests, 1 failures

My error was finally pointed out to me this morning on the elixir list and opened my eyes to a consistent flaw in my thinking about elixir. The fix of course is to actually provide a function to assert_raise, what I was actually providing was an expression that could return anything. 

 test "assert_raise works" do
    assert_raise(RuntimeError, fn -> RaiseTest.testraise end ) 
end

The key misunderstanding in my head was that function when used in the elixir context actually maps to what I think of as a function reference from my years of C programming. Module functions return expressions which can be anything. The test failed because it was calling TestRaisetest to see if it returned a function it could use in the test. 
When an elixir function requires a "function" as an argument, it really means a closure that can be executed.

Remember JSON is always valid YAML

While I don't mind YAML for relatively simple data files, I find it to be very difficult to use when there are more than one or two levels of nesting. If you have a relatively complex config file, then I find JSON to be a much easier to reason about and make any necessary changes.

For current versions of YAML, JSON files are a subset of the YAML standard. Thus any compliant YAML parser will also parson JSON.

The place where I use this trick the most often is in .kitchen.yml files used by TestKitchen when writing Chef cookbooks.  I start out by taking an existing .kitchen.yml file and converting it to
.kitchen.json using this website http://codebeautify.org/yaml-to-json-xml-csv. I then copy the
.kitchen.json to .kitchen.yml.

Writing Effective Examples in Elixir docstrings.

I had an email exchange on the elixir-core mailing list recently that I think is worth preserving as a blog post. It shows a great approach to how to write examples for functions with complex output.

We had a meetup this weekend for hacking the docs. The intent was to add examples to the standard library. 

I started looking around in code.ex and the question I run into is the following:

Should examples be functionally accurate or readable? 

Here's what I mean. I was attempting to add an example for 

Code.get_docs. 

This is what I came up with. 

## Examples

      iex> Code.get_docs(Atom, :docs)

      [{{:to_char_list, 1}, 36, :def, [{:atom, [], nil}],

      "Converts an atom to a char list.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n    iex> Atom.to_char_list(:\"An atom\")\\n    'An atom'\\n\\n"},

      {{:to_string, 1}, 20, :def, [{:atom, [], nil}],

      "Converts an atom to a string.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n    iex> Atom.to_string(:foo)\\n    \"foo\"\\n\\n"}]

      

      iex(1)> Code.get_docs(Atom, :all )

      [docs: [{{:to_char_list, 1}, 36, :def, [{:atom, [], nil}],

      "Converts an atom to a char list.\\n\\\nInlined by the compiler.\\n\\n## Examples\\n\\n    iex> Atom.to_char_list(:\"An atom\")\\n    'An atom'\\n\\n"},

      {{:to_string, 1}, 20, :def, [{:atom, [], nil}],

      "Converts an atom to a string.\\n\\nInlined by the compiler.\\n\\n## Examples\\n\\n    iex> Atom.to_string(:foo)\\n    \"foo\"\\n\\n"}],

      moduledoc: {1,"Convenience functions for working with atoms.\\n\\nSee also `Kernel.is_atom/1`.\\n"}]

 

It's really messy to read and I'm still working on getting the quoting correct for ex_doc. 

Atom is about the simplest standard module available, so picking anything else just 

gets worse. Also it suffers from the problem that it doesn't automatically update when

the docs for Atom update. 

My initial thought was to dynamically create a Sample module and then use Code.get_docs

on that example ( i.e. similar to the test for Code.get_docs ).  

So I guess the question is: 

Should any examples in the documentation exactly match the results from running the code in iex? 

i.e. while the standard lib does not use doctest as far as I know, any examples included should be

doctest aware. Or is it okay to simplify the results for clarity? 

- Booker C. Bense

José Valim

Feb 23

Should any examples in the documentation exactly match the results from running the code in iex?

Yes. But you can always "cheat"!

# Get the doc for the first function

iex> [fun|_] = Code.get_docs(Atom, :docs) |> Enum.sort()

# Each clause is in the format

iex> {{_function, _arity}, _line, _kind, _signature, text} = fun

# Let's get the first line of the text

iex> String.split(text, "\n") |> Enum.at!(0)

"NOW YOU HAVE A STRING"

So you are showing the whole process without printing it all the way.

This is a really nice way to take very complex dense output and recast it in a simple example that makes the return value of the function much clearer.