This document summarizes installation and use of BroControl, Bro’s interactive shell for operating Bro installations. Bro Control has two modes of operation: a stand-alone mode for managing a traditional, single-system Bro setup; and a cluster mode for maintaining a multi-system setup of coordinated Bro instances load-balancing the work across a set of independent machines. Below, we describe the installation process separately for the two modes. Once installed, the operation is pretty similar for both types; just keep in mind that if this documents refers to “nodes” and you’re in a stand-alone setup, there’s is only a single one and no worker/proxies.)
Contents
Download
You can find the latest BroControl release for download at http://www.bro-ids.org/download.
BroControl’s git repository is located at git://git.bro-ids.org/broctl. You can browse the repository here.
This document describes BroControl 1.0. See the CHANGES file for version history.
Prerequisites
Running BroControl requires the following prerequisites:
- A Unix system. FreeBSD, Linux, and MacOS are supported and should work out of the box. Note however that FreeBSD usually sees more testing as it is Bro’s primary development platform. Other Unix systems will quite likely require some tweaking. Note that in a cluster setup, all systems must be running exactly the same operating system.
- A version of Python >= 2.6.
- A bash (note in particular, that on FreeBSD, bash is not installed by default).
Installation
Stand-alone Bro
For installing a standalone Bro setup, just follow the Bro quickstart guide included with the distribution in doc/quickstart.rst.
Bro Cluster
A Bro Cluster is a set of systems jointly analyzing the traffic of a network link in a coordinated fashion. BroControl is able to operate such a setup from a central manager system pretty much transparently, hiding much of the complexity of the multi-machine installation.
A cluster consists of four types of components:
- Frontends.
- One or more frontends: Frontends load-balance the traffic across a set of worker machines.
- Worker nodes.
- Workers are doing the actual analysis, with each seeing a slice of the overall traffic as split by the frontend(s).
- One or more proxies.
- Proxies relay the communication between worker nodes.
- One manager.
- The manager provides the cluster’s user-interface for controlling and logging. During operation, the user only interacts with the manager; this is where BroControl is running.
For more information about the cluster architecture, including options for the frontend, see Bro’s cluster documentation included with the distribution in doc/cluster.rst. This document focuses on the installation of manager, workers, and the proxies. If not otherwise stated, in the following we use the terms “manager”, “worker”, and “proxy” to refer to Bro instances, not to physical machines; rather, we use the term “node” to refer to physical machines. There may be multiple Bro instances running on the same node. For example, it’s possible to run a proxy on the same node as the manager is operating on.
In the following, as an example setup, we will assume that our cluster consists of four nodes (not counting the frontend). The host names of the systems will be host1, host2, host3, and host4. We will configure the cluster so that host1 runs the manager and the (only) proxy, and host{2,3,4} are each running one worker. This is a typical setup, which will work well for many sites.
When installing a cluster, in addition to the prerequisites mentioned above, you need to
- have the same user account set up on all nodes. On the worker nodes, this user must have access to target network interface in promiscuous mode. ssh access from the manager node to this user account must be setup on all machines, and must work without asking for a password/passphrase.
- have some storage available on all nodes under the same path, which we will call the cluster’s prefix path. In the following, we will use /usr/local/bro as an example. The Bro user must be able to either create this directory or, where it already exists, must have write permission inside this directory on all nodes.
- have ssh and rsync installed.
With all prerequisites in place, perform the following steps to install a Bro cluster (as the Bro user) if you install from the Bro source code (which includes BroControl):
Configure and compile the Bro distribution using the cluster’s prefix path as —prefix:
> cd /path/to/bro/source/distribution > ./configure --prefix=/usr/local/bro && make && make install
Add <prefix>/bin to your PATH.
Create a cluster configuration file. There is an example provided, which you can edit according to the instructions in the file:
> cd /usr/local/bro > vi etc/broctl.cfg
Create a node configuration file to define where manager, workers, and proxies are to run. There is again an example, which defines the example scenario described above and can be edited as needed:
> cd /usr/local/bro > vi etc/node.cfg
Create a network configuration file that lists all of the networks which the cluster should consider as local to the monitored environment. Once, again the installation installs a template for editing:
> cd /usr/local/bro > vi etc/networks.cfg
Install workers and proxies using BroControl:
> broctl install
This installation process uses ssh and rdist to copy the configuration over to the remote machines so, as described above, you need to ensure that logging in via SSH works before the install will succeed.
Some tasks need to be run on a regular basis. On the manager node, insert a line like this into the crontab of the user running the cluster:
0-59/5 * * * * <prefix>/bin/broctl cron
Finally, you can start the cluster:
> broctl start
Getting Started
BroControl is an interactive interface to the cluster which allows you to, e.g., start/stop the monitoring or update its configuration. It is started with the broctl script and then expects commands on its command-line (alternatively, broctl can also be started with a single command directly on the shell’s command line):
> broctl Welcome to BroControl x.y Type "help" for help. [BroControl] >
As the message says, type help to see a list of all commands. We will now briefly summarize the most important commands. A full reference follows link:cmd_reference[below].
Once broctl.cfg and node.cfg are set up as described above, the monitoring can be started with the start command. In the cluster setup, this will successively start manager, proxies, and workers. The status command should then show all nodes as operating. To stop the monitoring, issue the stop command. exit leaves the shell.
On the manager system (and on the standalone system), you find the current set of (aggregated) logs in logs/current (which is a symlink to the corresponding spool directory.) The workers and proxies log into spool/proxy/ and spool/<worker-name>/, respectively. The manager/stand-alone logs are archived in logs/, by default once a day. Logs files of workers and proxies are discarded at the same rotation interval.
Whenever the BroControl configuration is modified in any way (including changes to configuration files and site-specific policy scripts), install installs the new version. No changes will take effect until install_ is run. Before you run install, check can be used to check for any potential errors in the new configuration, e.g., typos in scripts. If check does not report any problems, doing install will pretty likely not break anything.
Note that generally configuration changes only take effect after a restart of the affected nodes. The restart command triggers this. Some changes however can be put into effect on-the-fly without restarting any of the nodes by using the update command (again only after doing install first). Such dynamic updates generally work with all changes done which only modify const variables declared as redefinable (i.e., with Bro’s &redef attribute).
Generally, site-specific tuning needs to be done with local policy scripts, as in any Bro setup. This is described link:site_tuning[below].
BroControl provides various options to control the behavior of the setup. These options can be set by editing etc/broctl.cfg. The config command gives list of all options with their current values. A list of the most important options also follows link:cmd_reference[below].
Site-specific Customization
You’ll most likely want to adapt the Bro policy to the local environment and much of the more specific tuning requires writing local policy files.
By default, it is assumed that you put site-specific policy scripts into the share/bro/site directory. To change the location of site policies, set the option SitePolicyPath in broctl.cfg to a different path.
During the initial install, sample policy scripts are installed in share/bro/site, which you can edit as appropriate. In the stand-alone setup, this is just a single file called local.bro. In the cluster setup, there are three: local-manager.bro and local-worker.bro are loaded by the manager and the workers (plus proxies), respectively. In turn, both of these load local.bro, which contains all configuration code shared by all nodes. If in doubt, put your customizations into local.bro so that all nodes see it. If you want to change which local scripts are loaded by the nodes, you can set SitePolicyManager for the manager and SitePolicyWorker for the workers.
In the cluster setup, the main exception to putting everything into local.bro is notice filtering, which should be done only on the manager. The example local-manager.bro comes with an example setup to configure notice policy and notice actions. You should to adapt these to the local environment.
In general, all of BroControl‘s policy scripts are loaded before any site-specific policy so that you can redefine any of the defaults locally. The scripts shows precisely which policy scripts get loaded by a node; that can be very helpful for debugging.
It is also possible to add additional scripts to individual nodes only. This works by setting the option aux_scripts for the corresponding node(s) in etc/nodes.cfg. For example, one could add a script experimental.bro to a single worker for trying out new experimental code. Alternatively, the node tag can be used as a prefix as in the example worker-1.local.bro. The prefix will be automatically given to each node automatically by BroControl.
Command Reference
The following summary lists all commands supported by BroControl. All commands may be either entered interactively or specified on the shell’s command line. If not specified otherwise, commands taking [<nodes>] as arguments apply their action either to the given set of nodes, or to all nodes if none is given.
[?1034h.. Automatically generated. Do not edit.
- attachgdb [<nodes>]
- Primarily for debugging, the command attaches a gdb to the main Bro process on the given nodes.
- capstats [<interval>] [<nodes>]
Determines the current load on the network interfaces monitored by each of the given worker nodes. The load is measured over the specified interval (in seconds), or by default over 10 seconds. This command uses the capstats tool, which is installed along with broctl.
(Note: When using a CFlow and the CFlow command line utility is installed as well, the capstats command can also query the device for port statistics. TODO: document how to set this up.)
- check [<nodes>]
- Verifies a modified configuration in terms of syntactical correctness (most importantly correct syntax in policy scripts). This command should be executed for each configuration change before install is used to put the change into place. Note that check is the only command which operates correctly without a former install command; check uses the policy files as found in SitePolicyPath to make sure they compile correctly. If they do, install will then copy them over to an internal place from where the nodes will read them at the next start. This approach ensures that new errors in a policy scripts will not affect currently running nodes, even when one or more of them needs to be restarted.
- cleanup [—all] [<nodes>]
- Clears the nodes’ spool directories (if they are not running currently). This implies that their persistent state is flushed. Nodes that were crashed are reset into stopped state. If —all is specified, this command also removes the content of the node’s TmpDir, in particular deleteing any data potentially saved there for reference from previous crashes. Generally, if you want to reset the installation back into a clean state, you can first stop all nodes, then execute cleanup —all, and finally start all nodes again.
- config
- Prints all configuration options with their current values.
- cron [enable|disable|?] | [—no-watch] |
This command has two modes of operation. Without arguments (or just —no-watch), it performs a set of maintainance tasks, including the logging of various statistical information, expiring old log files, checking for dead hosts, and restarting nodes which terminated unexpectedly. The latter can be suppressed with the —no-watch option if no auto-restart is desired. This mode is intended to be run executed regularly via cron, as described in the installation instructions. While not intended for interactive use, no harm will be caused by executing the command manually: all the maintainance tasks will then just be performed one more time.
The second mode is for interactive usage and determines if the regular tasks are indeed performed when broctl cron is executed. In other words, even with broctl cron in your crontab, you can still temporarily disable its execution by running cron disable, and then later reenable with cron enable. This can be helpful while working, e.g., on the BroControl configuration and cron would interfere with that. cron ? can be used to query the current state.
- df [<nodes>]
- Reports the amount of disk space available on the nodes. Shows only paths relevant to the broctl installation.
- diag [<nodes>]
- If a node has terminated unexpectedly, this command prints a (somewhat cryptic) summary of its final state including excerpts of any stdout/stderr output, resource usage, and also a stack backtrace if a core dump is found. The same information is sent out via mail when a node is found to have crashed (the “crash report”). While the information is mainly intended for debugging, it can also help to find misconfigurations (which are usually, but not always, caught by the check command).
- exec <command line>
- Executes the given Unix shell command line on all nodes configured to run at least one Bro instance. This is handy to quickly perform an action across all systems.
- exit
- Terminates the shell.
- help
- Prints a brief summary of all commands understood by the shell.
- install
- Reinstalls the given nodes, including all configuration files and local policy scripts. This command must be executed after all changes to any part of the broctl configuration, otherwise the modifications will not take effect. Usually all nodes should be reinstalled at the same time, as any inconsistencies between them will lead to strange effects. Before executing install, it is recommended to verify the configuration with check.
- netstats [<nodes>]
- Queries each of the nodes for their current counts of captured and dropped packets.
- nodes
- Prints a list of all configured nodes.
- peerstatus [<nodes>]
- Primarily for debugging, peerstatus reports statistics about the network connections cluster nodes are using to communicate with other nodes.
- print <id> [<nodes>]
- Reports the current live value of the given Bro script ID on all of the specified nodes (which obviously must be running). This can for example be useful to (1) check that policy scripts are working as expected, or (2) confirm that configuration changes have in fact been applied. Note that IDs defined inside a Bro namespace must be prefixed with <namespace>:: (e.g., print SSH::did*ssh*version to print the corresponding table from ssh.bro.)
- process
- <trace> [options] [— scripts]]
Runs Bro offline on a given trace file using the same configuration as when running live. It does, however, use the potentially not-yet-installed policy files in SitePolicyPath and disables log rotation. Additional Bro command line flags and scripts can be given, the latter must be prefixed by —.
Upon completion, the command prints a path where the log files can be found. Subsequent runs of this command may delete these logs.
In cluster mode, Bro is run with both manager and worker scripts loaded into a single instance. While that doesn’t fully reproduce the live setup, it is often sufficient for debugging analysis scripts.
- quit
- Terminates the shell.
- restart [—clean] [<nodes>]
Restarts the given nodes, or all nodes if none are specified. The effect is the same as first executing stop followed by a start, giving the same nodes in both cases. This command is most useful to activate any changes made to Bro policy scripts (after running install first). Note that a subset of policy changes can also be installed on the fly via the update, without requiring a restart.
If —clean is given, the installation is reset into a clean state before restarting. More precisely, a restart —clean turns into the command sequence stop, cleanup —all, check, install, and start.
- scripts [-c] [<nodes>]
- Primarily for debugging Bro configurations, the script command lists all the Bro scripts loaded by each of the nodes in the order as they will be parsed by the node at startup. If -c is given, the command operates as check does: it reads the policy files from their original location, not the copies installed by install. The latter option is useful to check a not yet installed configuration.
- start [<nodes>]
- Starts the given nodes, or all nodes if none are specified. Nodes already running are left untouched.
- status [<nodes>]
- Prints the current status of the given nodes.
- stop [<nodes>]
- Stops the given nodes, or all nodes if none are specified. Nodes not running are left untouched.
- top [<nodes>]
- For each of the nodes, prints the status of the two Bro processes (parent process and child process) in a top-like format, including CPU usage and memory consumption. If executed interactively, the display is updated frequently until key q is pressed. If invoked non-interactively, the status is printed only once.
- update [<nodes>]
- After a change to Bro policy scripts, this command updates the Bro processes on the given nodes while they are running (i.e., without requiring a restart). However, such dynamic updates work only for a subset of Bro’s full configuration. The following changes can be applied on the fly: (1) The value of all const variables defined with the &redef attribute can be changed. More extensive script changes are not possible during runtime and always require a restart; if you change more than just the values of &redef-able consts and still issue update, the results are undefined and can lead to crashes. Also note that before running update, you still need to do an install (preferably after check), as otherwise update will not see the changes and resend the old configuration.
Option Reference
This section summarizes the options that can be set in etc/broctl.cfg for customizing the behaviour of BroControl. This section summarizes the options that can be set in etc/broctl.cfg for customizing the behavior of BroControl. Usually, one only needs to change the “user options”, which are listed first. The “internal options” are, as the name suggests, primarily used internally and set automatically. They are documented here only for reference.
User Options
- AuxPostProcessors (string, default _empty_)
- Additional log postprocessors, with paths separated by spaces.
- BroArgs (string, default _empty_)
- Additional arguments to pass to Bro on the command-line.
- CFlowAddr (string, default _empty_)
- If a cFlow load-balander is used, the address of the device (format: <ip>:<port>).
- CFlowPassword (string, default _empty_)
- If a cFlow load-balander is used, the password for accessing its configuration interface.
- CFlowUser (string, default _empty_)
- If a cFlow load-balander is used, the user name for accessing its configuration interface.
- CommTimeout (int, default 10)
- The number of seconds to wait before assuming Broccoli communication events have timed out.
- CompressLogs (bool, default 1)
- True to gzip archived log files.
- CronCmd (string, default _empty_)
- A custom command to run everytime the cron command has finished.
- CustomInstallBin (string, default _empty_)
- Additional executables to be installed into ${BinDir}, including full path and separated by spaces.
- Debug (bool, default 0)
- Enable extensive debugging output in spool/debug.log.
- HaveNFS (bool, default 0)
- True if shared files are mounted across all nodes via NFS (see FAQ).
- LogDir (string, default ”${BroBase}/logs”)
- Directory for archived log files.
- LogExpireInterval (int, default 0)
- Number of days log files are kept (zero means disabled).
- LogRotationInterval (int, default 3600)
- The frequency of log rotation in seconds for the manager/standalone node.
- MailAlarmPrefix (string, default “ALERT:”)
- Subject prefix for individual alerts triggered by Notice::ACTION_EMAIL.
- MailAlarms (bool, default 1)
- True if Bro should send mails for NOTICE_EMAIL alerts.
- MailAlarmsTo (string, default ”${MailTo}”)
- Destination address for broctl-generated alarm mails.
- MailFrom (string, default “Big Brother <bro@localhost>”)
- Originator address for broctl-generated mails.
- MailReplyTo (string, default _empty_)
- Reply-to address for broctl-generated mails.
- MailSubjectPrefix (string, default ”[Bro]”)
- General Subject prefix for broctl-generated mails.
- MailTo (string, default ”<user>”)
- Destination address for broctl-generated non-alarm mails. Default is to use the same address as +MailTo+.
- MakeArchiveName (string, default ”${BroBase}/share/broctl/scripts/make-archive-name”)
- Script to generate filenames for archived log files.
- MemLimit (string, default “unlimited”)
- Maximum amount of memory for Bro processes to use (in KB, or the string ‘unlimited’).
- MinDiskSpace (int, default 5)
- Percentage of minimum disk space available before warning is mailed.
- PFRINGClusterID (int, default @PF_RING_CLUSTER_ID@)
- If PF_RING flow-based load balancing is desired, this is where the PF_RING cluster id is defined. The default value is configuration-dependent and determined automatically by CMake at configure-time based upon whether PF_RING’s enhanced libpcap is available. Bro must be linked with PF_RING’s libpcap wrapper for this option to work.
- Prefixes (string, default “local”)
- Additional script prefixes for Bro, separated by colons. Use this instead of @prefix.
- SaveTraces (bool, default 0)
- True to let backends capture short-term traces via ‘-w’. These are not archived but might be helpful for debugging.
- SendMail (string, default ”@SENDMAIL@”)
- Location of the sendmail binary. Make this string blank to prevent email from being sent. The default value is configuration-dependent and determined automatically by CMake at configure-time
- SitePluginPath (string, default _empty_)
- Directories to search for custom plugins, separated by colons.
- SitePolicyManager (string, default “local-manager.bro”)
- Local policy file for manager.
- SitePolicyPath (string, default ”${PolicyDir}/site”)
- Directories to search for local policy files, separated by colons.
- SitePolicyStandalone (string, default “local.bro”)
- Local policy file for all Bro instances.
- SitePolicyWorker (string, default “local-worker.bro”)
- Local policy file for workers.
- StopTimeout (int, default 60)
- The number of seconds to wait before sending a SIGKILL to a node which was previously issued the ‘stop’ command but did not terminate gracefully.
- TimeFmt (string, default “%d %b %H:%M:%S”)
- Format string to print data/time specifications (see ‘man strftime’).
- TimeMachineHost (string, default _empty_)
- If the manager should connect to a Time Machine, the address of the host it is running on.
- TimeMachinePort (string, default “47757/tcp”)
- If the manager should connect to a Time Machine, the port it is running on (in Bro syntax, e.g., +47757/tcp+.
Internal Options
- BinDir (string, default ”${BroBase}/bin”)
- Directory for executable files.
- BroBase (string, default _empty_)
- Base path of broctl installation on all nodes.
- CapstatsPath (string, default ”${bindir}/capstats”)
- Path to capstats binary; empty if not available.
- CfgDir (string, default ”${BroBase}/etc”)
- Directory for configuration files.
- DebugLog (string, default ”${SpoolDir}/debug.log”)
- Log file for debugging information.
- HaveBroccoli (bool)
- True if Broccoli interface is available.
- HelperDir (string, default ”${BroBase}/share/broctl/scripts/helpers”)
- Directory for broctl helper scripts.
- LibDir (string, default ”${BroBase}/lib”)
- Directory for library files.
- LibDirInternal (string, default ”${BroBase}/lib/broctl”)
- Directory for broctl-specific library files.
- LocalNetsCfg (string, default ”${CfgDir}/networks.cfg”)
- File definining the local networks.
- LockFile (string, default ”${SpoolDir}/lock”)
- Lock file preventing concurrent shell operations.
- NodeCfg (string, default ”${CfgDir}/node.cfg”)
- Node configuration file.
- OS (string, default _empty_)
- Name of operation systems as reported by uname.
- PluginDir (string, default ”${LibDirInternal}/plugins”)
- Directory where standard plugins are located.
- PolicyDir (string, default ”${BroBase}/share/bro”)
- Directory for standard policy files.
- PolicyDirBroCtl (string, default ”${SpoolDir}/policy/broctl”)
- Directory for additioal broctl policy scripts.
- PolicyDirSiteInstall (string, default ”${SpoolDir}/policy/site”)
- Directory where the shell copies local policy scripts when installing.
- PolicyDirSiteInstallAuto (string, default ”${SpoolDir}/policy/auto”)
- Directory where the shell copies auto-generated local policy scripts when installing.
- PostProcDir (string, default ”${BroBase}/share/broctl/scripts/postprocessors”)
- Directory for log postprocessors.
- ScriptsDir (string, default ”${BroBase}/share/broctl/scripts”)
- Directory for executable scripts shipping as part of broctl.
- SpoolDir (string, default ”${BroBase}/spool”)
- Directory for run-time data.
- StandAlone (bool, default 0)
- True if running in stand-alone mode (see elsewhere).
- StateFile (string, default ”${SpoolDir}/broctl.dat”)
- File storing the current broctl state.
- StaticDir (string, default ”${BroBase}/share/broctl”)
- Directory for static, arch-independent files.
- StatsDir (string, default ”${LogDir}/stats”)
- Directory where statistics are kepts.
- StatsLog (string, default ”${SpoolDir}/stats.log”)
- Log file for statistics.
- TemplateDir (string, default ”${BroBase}/share/broctl/templates”)
- Directory where the *.in templates are copied into.
- Time (string, default _empty_)
- Path to time binary.
- TmpDir (string, default ”${SpoolDir}/tmp”)
- Directory for temporary data.
- TmpExecDir (string, default ”${SpoolDir}/tmp”)
- Directory where binaries are copied before execution.
- TraceSummary (string, default ”${bindir}/trace-summary”)
- Path to trace-summary script; empty if not available.
- Version (string, default _empty_)
- Version of the broctl.
Writing Plugins
BroControl provides a plugin interface to extend its functionality. A plugin is written in Python and can do any, or all, of the following:
- Perform actions before or after any of the standard BroControl commands is executed. When running before the actual command, it can filter which nodes to operate or stop the execution alltogether. When running after the command, it gets access to the commands success on a per-node basis (where applicable).
- Add custom commands to BroControl.
- Add custom options to BroControl defined in broctl.cfg.
- Add custom keys to nodes in defined in nodes.cfg.
- Add custom entries to the analysis command.
A plugin is written by deriving a new class from BroControl class Plugin. The Python script with the new plugin is then copied into a plugin directoriy searched by BroControl at startup. By default, BroControl searches <prefix>/lib/broctl/plugins; further may be configured by setting the PluginDir option. Note that any plugin script must end in *.py to be found. BroControl comes with two example plugins that can be used as a starting point; see <prefix>/lib/broctl/plugins/*.sample
In the following, we document the API that is available to plugins. A plugin must be derived from the Plugin class, and can use its methods as well as those of the Node class.
Class Plugin
- class Plugin
The class Plugin is the base class for all BroControl plugins.
The class has a number of methods for plugins to override, and every plugin must at least override name() and pluginVersion().
For each BroControl command foo, there’s are two methods, cmd_foo_pre and cmd_foo_post, that are called just before the command is executed and just after it has finished, respectivey. The arguments these methods receive correspond to their command-line parameters, and are further documented belows.
The cmd_<XXX>_pre methods have the ability to prevent the command’s execution, either completely or partially for those commands that take nodes as parameters. In the latter case, the method receives a list of nodes that the command is to be run on, and it can filter that list and returns modified version of nodes actually to use. The standard case would be returning simply the unmodified nodes parameter. To completely block the command’s execution, return an empty list. To just not execute the command for a subset, remove them affected ones. For commands that do not receive nodes as arguments, the return value is interpreted as boolean indicated whether command execution should proceed (True) or not (False).
The cmd_<XXX>_post methods likewise receive the commands arguments as their parameter, as documented below. For commands taking nodes, the list corresponds to those nodes for which the command was actually executed (i.e., after any cmd_<XXX>_pre filtering). Each node is given as a tuple (node, bool) with node being the actual Node, and the boolean indicating whether the command was succesful for it.
Note that if plugin prevents a command from execution either completely or partially, it should report its reason via the message*( or error() methods.
If multiple plugins hook into the same command, all their cmd_<XXX>_{pre,post} are executed in undefined order. The command is executed on the intersection of all cmd_<XXX>_pre results.
Finally, note that the restart command doesn’t have its own method as it’s just a combination of other commands and thus their callbacks are run.
debug (self, msg)
Logs a debug message in BroControl’ debug log if enabled.
error (self, msg)
Reports an error to the user.
execute (self, node, cmd)
Executes a command on the host for the given node of type Node. Returns a tuple (success, output) in which succes is True if the command run successfully and output the combined stdout/stderr output.
executeParallel (self, cmds)
Executes a set of commands in parallel on multiple hosts. cmds is a list of tuples (node, cmd), in which the node is Node instance and cmd a string with the command to execute for it. The method returns a list of tuples (node, success, output), in which success is True if the command run successfully and output the combined stdout/stderr output for the corresponding node.
getGlobalOption (self, name)
Returns the value of the global BroControl option or state attribute name. If the user has not set the options, its default value is returned. See the output of broctl config for a complete list
getOption (self, name)
Returns the value of one of the plugin’s options, name. The returned value will always be a string.
An option has a default value (see options()), which can be overridden by a user in broctl.cfg. An option’s value cannot be changed by the plugin.
getState (self, name)
Returns the current value of one of the plugin’s state variables, name. The returned value will always be a string. If it has not yet been set, an empty string will be returned.
Different from options, state variables can be set by the plugin and are persistent across restarts. They are not visible to the user.
Note that a plugin cannot query any global BroControl state variables.
hosts (self, nodes)
Returns a list of all hosts running at least one node from the list of Nodes objects in nodes, or configured in if nodes is empty.
message (self, msg)
Reports a message to the user.
nodes (self)
Returns a list of all configured Node objects.
parseNodes (self, names)
Returns Node objects for a string of space-separated node names. If a name does not correspond to a known node, an error message is printed and the node is skipped from the returned list. If no names are known, an empty list is returned.
setState (self, name, value)
Sets the one of the plugin’s state variables, name, to value. value must be a string. The change is permanent and will be recorded to disk.
Note that a plugin cannot change any global BroControl state variables.
broProcessDied (self, node)
Called when BroControl finds the Bro process for Node node to have terminated unexpectedly. This method will be called just before BroControl prepare the node’s “crash report” and before it cleans up the node’s spool directory.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_attachgdb_post (self, nodes)
- Called just after the attachgdb command has finished. Arguments
- are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_attachgdb_pre (self, nodes)
Called just before the attachgdb command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_capstats_post (self, nodes, interval)
Called just after the capstats command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_capstats_pre (self, nodes, interval)
Called just before the capstats command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. integer is an integer with the measurement interval in seconds.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_check_post (self, results)
Called just after the check command has finished. It receives the list of 2-tuples (node, bool) indicating the nodes the command was executed for, along with their success status.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_check_pre (self, nodes)
Called just before the check command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_cleanup_post (self, nodes, all)
Called just after the cleanup command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_cleanup_pre (self, nodes, all)
Called just before the cleanup command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. all is boolean indicating whether the —all argument has been given.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_config_post (self)
Called just after the config command has finished.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_config_pre (self)
Called just before the config command is run.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_cron_post (self, arg, watch)
Called just after the cron command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_cron_pre (self, arg, watch)
Called just before the cron command is run. arg is None if the cron is executed without arguments. Otherwise, it is one of strings enable, disable, ?. watch is a boolean indicating whether cron should restart abnormally terminated Bro processes; it’s only valid if arg is empty.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_custom (self, cmd, args)
Called when command defined by the commands method is executed. cmd is the command (with the plugin’s prefix), and args is a single string with all arguments.
If the arguments are actually node names, parseNodes can be used to get the Node objects.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_df_post (self, nodes)
- Called just after the df command has finished. Arguments are as
- with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_df_pre (self, nodes)
Called just before the df command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_diag_post (self, nodes)
Called just after the diag command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_diag_pre (self, nodes)
Called just before the diag command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_exec_post (self, cmdline)
Called just after the exec command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_exec_pre (self, cmdline)
Called just before the exec command is run. cmdline is a string with the command line to execute.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_install_post (self)
Called just after the install command has finished.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_install_pre (self)
Called just before the install command is run.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_netstats_post (self, nodes)
- Called just after the netstats command has finished. Arguments
- are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_netstats_pre (self, nodes)
Called just before the netstats command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_nodes_post (self)
Called just after the nodes command has finished.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_nodes_pre (self)
Called just before the nodes command is run.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_peerstatus_post (self, nodes)
Called just after the peerstatus command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_peerstatus_pre (self, nodes)
Called just before the peerstatus command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_print_post (self, nodes, id)
Called just after the print command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_print_pre (self, nodes, id)
Called just before the print command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. is is a string with the name of the ID to printed.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_process_post (self, trace, options, scripts, success)
Called just after the process command has finished. Arguments are as with the pre method, plus an additional boolean success indicating whether Bro terminated normally.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_process_pre (self, trace, options, scripts)
Called just before the process command is run. It receives the trace to read from as a string, a list of additional Bro options, and a list of additional Bro scripts.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_restart_post (self, results)
Called just after the restart command has finished. It receives the list of 2-tuples (node, bool) indicating the nodes the command was executed for, along with their success status. The remaining arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_restart_pre (self, nodes, clean)
Called just before the restart command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. clean is boolean indicating whether the —clean argument has been given.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_scripts_post (self, nodes, full_path, check)
Called just after the scripts command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_scripts_pre (self, nodes, full_path, check)
Called just before the scripts command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. full_path and check are boolean indicating whether the -p and -c options were given, respectively.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_start_post (self, results)
Called just after the start command has finished. It receives the list of 2-tuples (node, bool) indicating the nodes the command was executed for, along with their success status.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_start_pre (self, nodes)
Called just before the start command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_status_post (self, nodes)
Called just after the status command has finished. Arguments are as with the pre method.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_status_pre (self, nodes)
Called just before the status command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_stop_post (self, results)
Called just after the stop command has finished. It receives the list of 2-tuples (node, bool) indicating the nodes the command was executed for, along with their success status.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_stop_pre (self, nodes)
Called just before the stop command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_top_post (self, nodes)
Called just after the top command has finished. Arguments are as with the pre method. Note that when top is run interactively to auto-refresh continously, this method will be called once after each update.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_top_pre (self, nodes)
Called just before the top command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command. Note that when top is run interactively to auto-refresh continously, this method will be called once before each update.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_update_post (self, results)
Called just after the update command has finished. It receives the list of 2-tuples (node, bool) indicating the nodes the command was executed for, along with their success status.
This method can be overridden by derived classes. The default implementation does nothing.
cmd_update_pre (self, nodes)
Called just before the update command is run. It receives the list of nodes, and returns the list of nodes that should proceed with the command.
This method can be overridden by derived classes. The default implementation does nothing.
commands (self)
Returns a set of custom commands provided by the plugin.
The return value is a list of 2-tuples each having the following elements:
- command
- A string with the command’s name. Note that command name exposed to the user will be prefixed with the plugin’s prefix as returned by name() (e.g., myplugin.mycommand).
- arguments
- A string describing the command’s arguments in a textual form suitable for use in the help command summary (e.g., [<nodes>] for command taking an optional list of nodes). Empty if no arguments are expected.
- description
- A string with a description of the command’s semantics.
This method can be overridden by derived classes. The implementation must not call the parent class’ implementation. The default implementation returns an empty list.
done (self)
Called once just before BroControl terminates. This method can do any cleanup the plugin may require.
This method can be overridden by derived classes. The default implementation does nothing.
hostStatusChanged (self, host, status)
Called when BroControl’s cron command finds the availability of a cluster system to have changed. Initially, all systems are assumed to be up and running. Once BroControl notices that a system isn’t responding (defined as either it doesn’t ping at all, or does not accept SSH sessions), it calls this method, passing in a string with the name of the host and a boolean status set to False. Once the host becomes available again, the method will be called again for the same host with status now set to True.
Note that BroControl’s cron tracks a hosts availability across execution, so if the next time its run the host is still down, this method will be not called again.
This method can be overridden by derived classes. The default implementation does nothing.
init (self)
Called once just before BroControl starts executing any commands. This method can do any initialization that the plugin may require.
Note that at when this method executes, BroControl guarantees that all internals are fully set up (e.g., user-defined options are available). This may not be the case when the class __init__ method runs.
Returns a boolean, indicating whether the plugin should be used. If it returns False, the plugin will be removed and no other methods called.
This method can be overridden by derived classes. The default implementation always returns True.
name (self)
Returns a a strings with a descriptive name for the plugin (e.g., “TestPlugin”). The name must not contain any white-space.
This method must be overridden by derived classes. The implementation must not call the parent class’ implementation.
nodeKeys (self)
Returns a list of custom keys for node.cfg. The value for a keys will be available from the Node object as attribute <prefix>_<key> (e.g., node.test_mykw). If not set, the attribute will be set to None.
This method can be overridden by derived classes. The implementation must not call the parent class’ implementation. The default implementation returns an empty list.
options (self)
Returns a set of local configuration options provided by the plugin.
The return value is a list of 4-tuples each having the following elements:
- name
- A string with name of the option (e.g., Path). Option names are case-insensitive. Note that the option name exposed to the user will be prefixed with your plugin’s prefix as returned by name() (e.g., myplugin.Path).
- type
- A string with type of the option, which must be one of “bool”, “string”, or “int”.
- default
- A string with the option’s default value. Note that this must always be a string, even for non-string types. For booleans, use “0” for False and “1” for True. For integers, give the value as a string “42”.
- description
- A string with a description of the option semantics.
This method can be overridden by derived classes. The implementation must not call the parent class’ implementation. The default implementation returns an empty list.
pluginVersion (self)
Returns an integer with a version number for the plugin. Plugins should increase their version number with any significant change.
This method must be overridden by derived classes. The implementation must not call the parent class’ implementation.
prefix (self)
Returns a string with a prefix for the plugin’s options and commands names (e.g., “myplugin”)“).
This method can be overridden by derived classes. The implementation must not call the parent class’ implementation. The default implementation returns a lower-cased version of name().
Class Node
- class Node
Class representing on node of the BroControl maintained setup. In standalone mode, there’s always exactly one node of type standalone`. In a cluster setup, there is exactly one of type “manager, and zero or more of type proxy and worker.
In addition to the methods described above, a Node object has a number keys with values that are set via nodes.cfg and can be accessed directly via corresponding Python attributes (e.g., node.name):
- name (string)
- The name of the node, which corresponds to the [<name>] section in nodes.cfg.
- type (string)
- The type of the node, which will be one of standalone, manager, proxy, and worker.
- host (string)
- The hostname of the system the node is running on.
- interface (string)
- The network interface for Bro to use; empty if not set.
- aux_scripts (string)
- Any node-specific Bro script configured for this node.
Any attribute that is not defined in nodes.cfg will be empty.
In addition, plugins can override Plugin.nodeKeys to define their own node keys, which can then be likewise set in nodes.cfg. They key names will be prepended with the plugin’s Plugin.prefix (e.g., for the plugin test, the node key foo is set by adding test.foo=value to node.cfg.
cwd (self)
Returns a string with node’s working directory.
describe (self)
Returns an extended string representation of the node including all its keys with values.
getPID (self)
Returns the process ID of the node’s Bro process if running, and None otherwise.
getPort (self)
Returns an integer with the port that this node’s communication system is listening on for incoming connections, or -1 if no such port has been set yet.
hasCrashed (self)
Returns True if the node’s Bro process has exited abnormally.
Miscellanous
Mails
BroControl sends four types of mails to the address given in MailTo:
- When logs are rotated (per default once a day), a list of all alarms during the last rotation interval is sent. This can be disabled by setting MailAlarms=0.
- When the cron command notices that a node has crashed, it restarts it and sends a notification. It may also send a more detailed crash report containing information about the crash.
- NOTICES with a notice action EMAIL.
- If link:trace_summary[trace-summary] is installed, a traffic summary is sent each rotation interval.
Performance Analysis
TODO: broctl cron logs a number of statistics, which can be analyzed/plotted for understanding the clusters run-time behavior.
Questions and Answers
- Can I use an NFS-mounted partition as the cluster’s base directory to avoid the “rsync“’ing?
- Yes. BroBase can be on an NFS partition. Configure and install the shell as usual with —prefix=<BroBase>. Then add HaveNFS_=1+ and SpoolDir=<spath> to etc/broctl.cfg, where <spath> is a path on the local disks of the nodes; <spath> will be used for all non-shared data (make sure that the parent directory exists and is writable on all nodes!). Then run xref:cmd_install[+make install-broctl+] again. Finally, you can remove <BroBase>/spool (or link it to <spath>). In addition, you might want to keep the log files locally on the nodes as well by setting LogDir_+ to a non-NFS directory. (Only the manager’s logs will be kept permanently, the logs of workers/proxies are discarded upon rotation.)
- When I’m using the standalone mode_, do I still need to have “ssh“ and “rsync“ installed and configured?
- No. The standalone performs all operations directly on the local file system.
- What do I need to do when the something in the Bro distribution changes?
- After pulling from the main Bro git repository, just re-run make install inside your build directory. It will reinstall all the files from the distribution that are not up-to-date. Then do broctl install to make sure everything gets pushed out.
- Can I change the naming scheme that BroControl uses for archived log files?
- Yes, set MakeArchiveName_+ to a script that outputs the desired destination file name for an archived log file. The default script for that task is <BroBase>/share/broctl/scripts/make-archive-name, which you can use that as a template for creating your own version. See the beginning of that script for instructions.
