Generating Script Documentation

This is an overview of how we use Sphinx for documenting Bro scripts.

Sphinx does not auto-extract information, like Doxygen does, but that’s not much of a problem because it wouldn’t be able to parse Bro scripts anyway. Instead, we auto-generate input for Sphinx from Bro scripts, using Bro itself. For each script, Bro prints out a page suitably formatted for Sphinx based on all the relevant information it has extracted from the script.

A demonstration of how this looks with a (contrived) example Bro script:

The example.bro source script comes with ##-stylized comments embedded, to indicate what is supposed to be extracted for documentation purposes.

Then Bro, run in a new “documentation mode”, converts this script into example.rst, a reStructuredText input file for Sphinx.

Finally, Sphinx transforms that into example.html.

Makefile targets restdoc and doc are available to automate the generation process of reST and HTML documentation for all scripts, respectively. In the Bro source distribution, see doc/scripts/README for more info.

Notes:

All information in the final HTML is found in the original script; some pieces are just part of the normal script code (like the packet filter), while others are provided manually in the ## comments.
A basic assumption is that everything export‘d from a namespace (module) or declared in the GLOBAL namespace gets documented as part of the script’s public API regardless of whether there’s a comment attached.
The custom Bro-directives/roles for Sphinx shown in the example (e.g. .. bro:id:, :bro:type:) are added to Sphinx via its plugin mechanism in a “Bro domain”. These roles can be directly embedded in the original Bro script’s ##-stylized comments as a way of adding a cross-referencing link in the final HTML result:
- :bro:id:
- :bro:type:
- :bro:enum:
- :bro:attr:
- :bro:namespace:
Note that although identifiers, types, and enums have some relation in Bro scripting-land, in documentation-land they are distinct. That is, in Bro scripting, an enum is a special case of type and a declared type also has an identifier associated with it, but when documenting them, the correct, most-specific reST role has to be used (e.g. :bro:id: will not work when referencing declared types, use :bro:type: instead, and use :bro:enum: if that type is actually an enum ). Example usage:
```
##! This links to the documentation for a variable that can
##! be declared in any Bro script: :bro:id:`local_zones`
```
Such a custom domain allows for extensive cross-referencing (e.g., whenever conn_id is used, it would be linked to the definition of that record); and indexing (e.g., we can have comprehensive indices of events, functions, notices, scripts, etc.)

There is also a very useful sphinx role :doc:, that can be used to link to another script:

##! This renders a link (in the summary section) to the
##! documentation for conn's main.bro:
##! :doc:`/scripts/base/protocols/conn/main`
##!
##! And this would render a link to an index of all the script docs
##! in that base/protocols/conn directory:
##! :doc:`/scripts/base/protocols/conn/index`