Upgrading From Bro 1.5 to 2.0

This guide details differences between Bro versions 1.5 and 2.0 that may be important for users to know as they work on updating their Bro deployment/configuration to the later version.

Contents

• Upgrading From Bro 1.5 to 2.0
  • Introduction
  • Default Scripts
    • Organization
    • Logging Framework
    • Notice Framework
    • New Default Settings
    • API Changes
    • Variable Naming
  • BroControl
  • Removed Functionality
  • Development Infrastructure

Introduction

As the version number jump suggests, Bro 2.0 is a major upgrade and lots of things have changed. Most importantly, we have rewritten almost all of Bro’s default scripts from scratch, using quite different structure now and focusing more on operational deployment. The result is a system that works much better “out of the box”, even without much initial site-specific configuration. The down-side is that 1.x configurations will need to be adapted to work with the new version. The two rules of thumb are:

1. If you have written your own Bro scripts that do not depend on any of the standard scripts formerly found in policy/, they will most likely just keep working (although you might want to adapt them to use some of the new features, like the new logging framework; see below).
2. If you have custom code that depends on specifics of 1.x default scripts (including most configuration tuning), that is unlikely to work with 2.x. We recommend to start by using just the new scripts first, and then port over any customizations incrementally as necessary (they may be much easier to do now, or even unnecessary). Send mail to the Bro user mailing list if you need help.

Below we summarize changes from 1.x to 2.x in more detail. This list isn’t complete, see the CHANGES file in the distribution for the full story.

Default Scripts

Organization

In versions before 2.0, Bro scripts were all maintained in a flat directory called policy/ in the source tree. This directory is now renamed to scripts/ and contains major subdirectories base/, policy/, and site/, each of which may also be subdivided further.

The contents of the new scripts/ directory, like the old/flat policy/ still gets installed under under the share/bro subdirectory of the installation prefix path just like previous versions. For example, if Bro was compiled like ./configure --prefix=/usr/local/bro && make && make install, then the script hierarchy can be found in /usr/local/bro/share/bro.

THe main subdirectories of that hierarchy are as follows:

• base/ contains all scripts that are loaded by Bro by default (unless the -b command line option is used to run Bro in a minimal configuration). Note that is a major conceptual change: rather than not loading anything by default, Bro now uses an extensive set of default scripts out of the box.

  The scripts under this directory generally either accumulate/log useful state/protocol information for monitored traffic, configure a default/recommended mode of operation, or provide extra Bro scripting-layer functionality that has no significant performance cost.

• policy/ contains all scripts that a user will need to explicitly tell Bro to load. These are scripts that implement functionality/analysis that not all users may want to use and may have more significant performance costs. For a new installation, you should go through these and see what appears useful to load.

• site/ remains a directory that can be used to store locally developed scripts. It now comes with some preinstalled example scripts that contain recommended default configurations going beyond the base/ setup. E.g. local.bro loads extra scripts from policy/ and does extra tuning. These files can be customized in place without being overwritten by upgrades/reinstalls, unlike scripts in other directories.

With version 2.0, the default BROPATH is set to automatically search for scripts in policy/, site/ and their parent directory, but not base/. Generally, everything under base/ is loaded automatically, but for users of the -b option, it’s important to know that loading a script in that directory requires the extra base/ path qualification. For example, the following two scripts:

• $PREFIX/share/bro/base/protocols/ssl/main.bro
• $PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro

are referenced from another Bro script like:

@load base/protocols/ssl/main
@load protocols/ssl/validate-certs

Notice how policy/ can be omitted as a convenience in the second case. @load can now also use relative path, e.g., @load ../main.

Logging Framework

• The logs generated by scripts that ship with Bro are entirely redone to use a standardized, machine parsable format via the new logging framework. Generally, the log content has been restructured towards making it more directly useful to operations. Also, several analyzers have been significantly extended and thus now log more information. Take a look at ssl.log.
  • A particular format change that may be useful to note is that the conn.log service field is derived from DPD instead of well-known ports (while that was already possible in 1.5, it was not the default).
  • Also, conn.log now reports raw number of packets/bytes per endpoint.
• The new logging framework makes it possible to extend, customize, and filter logs very easily. See the logging framework more information on usage.
• A common pattern found in the new scripts is to store logging stream records for protocols inside the connection records so that state can be collected until enough is seen to log a coherent unit of information regarding the activity of that connection. This state is now frequently seen/accessible in event handlers, for example, like c$<protocol> where <protocol> is replaced by the name of the protocol. This field is added to the connection record by redef‘ing it in a base/protocols/<protocol>/main.bro script.
• The logging code has been rewritten internally, with script-level interface and output backend now clearly separated. While ASCII logging is still the default, we will add further output types in the future (binary format, direct database logging).

Notice Framework

The way users interact with “notices” has changed significantly in order to make it easier to define a site policy and more extensible for adding customized actions. See the notice framework.

New Default Settings

• Dynamic Protocol Detection (DPD) is now enabled/loaded by default.
• The default packet filter now examines all packets instead of dynamically building a filter based on which protocol analysis scripts are loaded. See PacketFilter::all_packets for how to revert to old behavior.

API Changes

• The @prefixes directive works differently now. Any added prefixes are now searched for and loaded after all input files have been parsed. After all input files are parsed, Bro searches BROPATH for prefixed, flattened versions of all of the parsed input files. For example, if lcl is in @prefixes, and site.bro is loaded, then a file named lcl.site.bro that’s in BROPATH would end up being automatically loaded as well. Packages work similarly, e.g. loading protocols/http means a file named lcl.protocols.http.bro in BROPATH gets loaded automatically.
• The make_addr BIF now returns a subnet versus an addr

Variable Naming

• Module is more widely used for namespacing. E.g. the new site.bro exports the local_nets identifier (among other things) into the Site module.
• Identifiers may have been renamed to conform to new scripting conventions

BroControl

BroControl looks pretty much similar to the version coming with Bro 1.x, but has been cleaned up and streamlined significantly internally.

BroControl has a new process command to process a trace on disk offline using a similar configuration to what BroControl installs for live analysis.

BroControl now has an extensive plugin interface for adding new commands and options. Note that this is still considered experimental.

We have remove the analysis command, and BroControl does currently not not send daily alarm summaries anymore (this may be restored later).

Removed Functionality

We have remove a bunch of functionality that was rarely used and/or had not been maintained for a while already:

• The net script data type.
• The alarm statement; use the notice framework instead.
• Trace rewriting.
• DFA state expiration in regexp engine.
• Active mapping.
• Native DAG support (may come back eventually)
• ClamAV support.
• The connection compressor is now disabled by default, and will be removed in the future.

Development Infrastructure

Bro development has moved from using SVN to Git for revision control. Users that like to use the latest Bro developments by checking it out from the source repositories should see the development process. Note that all the various sub-components now reside on their own repositories. However, the top-level Bro repository includes them as git submodules so it’s easu to check them all out simultaneously.

Bro now uses CMake for its build system so that is a new required dependency when building from source.

Bro now comes with a growing suite of regression tests in testing/.