The new Match module is a flexible text-stream scanner using regular-expression matching. Its regular expressions are similar to those of 'grep' (the Shell Tool), but it also allows multi-line, arbitrary length matches, is UTF8-aware, and can reformat the matches it extracts.
A couple of general utility modules are included: 'JoinPaths' lets you merge data streams, and 'Switcher' lets you select the path you wish data to take.
The immediately following section, 'Diagrams' shows how configurations are built in a Weaver window. After that, 'Elements and Connectors' goes into more detail about the components, and I end the overview with some comments on the scheme.
Everything you need to run the StreamWeaver is in this ('Weaver') folder. Here you will find the Weaver program itself and all the modules (most of which are in the 'StreamWeaver' sub-folder), plus a ready-built example diagram. As a quick start, double-click on this example. the 'simplestream' icon: this will start up the Weaver and load the example diagram.
'simplestream' is about as simple as you can get. The diagram has a 'Source' StreamView on the left feeding a PipeStream element (middle) that is just running a 'cat' command. The output from this goes to the 'Output' StreamView on the right. As 'cat' simply copies everything from its input to its output, anything you type into 'Source' appears in 'Output'. Experiment by typing something into 'Source', see it echoed in 'Output'. Then select part of what you typed, and click the 'SEND Selected' button.
Click 'SEND End-Of-Data', and notice how this gets passed on through 'cat' to the output. The 'cat' command is terminated by this, but because 'Auto-Restart' is selected in the PipeStream, it immediately gets restarted, ready for more input.
The best way to learn about building and modifying diagrams is to add a little bit to this one. In the top 'Weaver' drawer there is a module called 'JoinPaths'. With the mouse, grab the icon and drag it directly on top of the path between the 'Source' (left-hand) and 'Command' (centre) elements -- somewhere in the middle, not too close to either end. You should see a new element inserted into the path. Next, open the 'StreamWeaver' folder itself, and similarly drop a ReadFile element into the diagram, by dragging its icon from the folder; put it somewhere above and to the left of the JoinPaths. Note that as ReadFile is a 'Source', not a 'Filter', you can't insert it into a path as you did the JoinPaths. Any time after it is placed, you can move an element around if you wish by grabbing its centre-point with the mouse and dragging to a new position.
Connect the ReadFile to the JoinPaths by dragging from the output connector (the little pip on the right side) of the ReadFile to the top side of the JoinPaths. A path from one to the other should appear. Now you have two alternate sources for the PipeStream: the original StreamView and the added ReadFile.
To use the ReadFile, you need to open its 'Control Panel'. (The ones for the original three elements were thoughtfully opened for you already...) If you have at least a two-button mouse, depress the right button over the centre of the ReadFile element in the diagram; if you don't have the second button, hold down <Ctrl> or <Shift> while you press the mouse button. You will get a short popup menu (plus an info window). Still holding the button down, select the top 'Panel' item and release. The small control panel for the element should appear. To read a file (which should be text for convenience, but doesn't have to be in general) you can either click on the 'Read...' button, which will open up a standard File Selection Panel where you can choose your file, or simply drag the file onto the control panel (not the File Selection Panel). If you want to drag a file, a convenient one is 'simplestream' itself: configuration files are just text. [Note that in this release (1.1), unlike the previous, all module panels respond to the first click of the mouse.]
We've just converged two paths into one. It is even easier to split one path into two. Let's add a 'WriteFile' element to record the output from the PipeStream. Drag the WriteFile icon to somewhere below and to the right of the PipeStream. Then press the mouse button exactly on the path between the latter and the 'Output' StreamView, and drag a new -- branch -- path from that to the WriteFile (top or left side). Output data will now flow both to the StreamView and the WriteFile. To actually record to a file, you have to specify one, so open the new element's control panel as before, and click on 'Write to...' to open a standard FileSave Panel. (As WriteFile is a simple 'Filter' element that can pass incoming data through without any modification, you could have simply inserted it into the output path itself, without branching, but I wanted to illustrate that operation.)
Try playing around further with the 'Command' element. Replace 'cat' with 'sort', say, and send some test lines or a complete file. (Remember that you won't see anything out of 'sort' until it gets an End-Of-Data.). Change the mode to see what happens, and so on.
After setting up your diagram, you perhaps want to save it. The menu to do this is brought up by pressing the right mousebutton over any blank area of the diagram. Select 'Save as...' and follow the usual procedure. (Avoid choosing the 'Save' item above it unless you really want to overwrite the original 'simplestream' diagram, of course.) The 'Diagram Name' item of the same menu lets you change the label in the diagram's window tab, which can be helpful if you have several open at once; the name will remain attached to the diagram if you save it.
Finally, the -- so far ignored -- 'Weaver Main' window at upper left of the screen just has a menu with two relevant functions: 'QUIT' to close everything down, and 'New Diagram', which brings up a new blank slate (existing windows are not affected).
This has been a fairly brief run-through. For a rather fuller discussion of the Weaver and its features, you might want to take a look at the documentation in the MusicWeaver, which uses the same scheme for real-time enhancement of MIDI. You should be able to find this in the same place from which you obtained the StreamWeaver.
For both PipeStream and StreamView, one of the two outputs supplies the actual data originating at the element (command output or user-entered text respectively). The other simply passes on all input data, so that you can chain inputs to multiple elements if you desire. By default the first output connection made will be the original data, a second will be the pass-through. You can rearrange this if you desire by pressing the right mousebutton over the (connected) output pip in question: a menu of the possible choices will pop up, and you can select which you want (any other current connections will readjust appropriately).
In contrast, 'WriteFile' and 'JoinPaths' are 'Filter' elements, which always only have one input and one output. However, unlike a multiconnector element, the input can merge up to three paths, which can be attached to the three non-output sides of the element. Typically, a filter performs some operation on its input and passes the result to its output. In the particular cases of both JoinPaths (which does nothing) and WriteFile (which just copies the data to a file), the output is identical to the input, but this is not in general so for filters in other Weaver environments.
'ReadFile' is a 'Source element, which has no inputs at all. You can attach a path to the output connector only: attempts to connect elsewhere will be refused.
I have found situations where the scheme is quite useful, however. There is convenience in being able to keep a visual history of one's input to command-line programs, that can be easily re-sent and re-edited. For instance, I now typically experiment with GNUplot and RLab using the 'simplestream' configuration, because it is so easy to change and re-try blocks of commands and data.
There was one occasion when I was trying to find what characters in a local provider's web page were hanging up Netscape 4 under Windows. I grabbed the source, fed it to a StreamView, easily found the rectangles indicating those characters, selected these, and sent them to 'od'... [Turned out they were simply massive numbers of returns. Oh well.]
There are some annoying, but unavoidable, limitiations on what one might otherwise like to do, simply because of the way shell-based programs themselves handle data. They typically buffer their generated output, only sending it on when either the buffer fills or the program ends. This means that you can't, for example, feed text line by line to 'grep' and expect to see any matched lines on the output immediately. This inability to see or detect 'events' as they happen can be very frustrating.
More modules can, and probably will, be added to the basic set. I actually see these modules more as a bridge from, say, the MusicWeaver to the shell environment. The Weaver program itself is common to all the individual application suites: only the type of data flowing changes. Modules simply ignore data they are not equipped to handle, so different suites could be intermixed even in the same diagram. (Although data within the StreamWeaver is pretty much a generic byte stream, it is still specific to these modules, and does not affect any others.)
There is a module (not yet distributed but probably available soon with the MusicWeaver) that translates a MusicWeaver MIDI stream into simple ASCII data, which could then be fed to, say, a user-written Python or Perl script. Another module could turn output from this back into MIDI. It should be a nice way to do customized processing (though probably not at high speed...).
Notice that this is a 'Multiconnector' element. The first output path connected is by default the command's standard output, but a second one may also be attached to get a 'pass-through' of the input stream.
The element control panel has two text entry fields: one for the desired execution directory, and the other for the command itself. The directory field is unimportant unless the command accesses or creates files (or unless the command itself is not in the default shell path); if you leave it blank, it will be taken as the system default -- presumably /boot/home. (If you type in an invalid directory path, an alert box will appear to notify you, but the name is retained; you can create the directory -- via the Tracker or otherwise -- before starting the command if you wish. Changing the directory while a command is running has no effect until a new run is started.)
Typing Enter after typing the command line will start execution immediately (and the displayed status will change from 'Stopped' to 'RUNNING'). You can also use the 'Run' button to begin execution. You have to use this if you want to run the same command again, because the text-entry control ignores the Enter key if the line has not been changed.
Depending on the command you enter, it may finish its own execution and terminate, or it may terminate when it receives 'end-of-data' on its input, or it might just keep on going until forcibly killed. If you need to kill it, the 'Stop' button should do the job -- but see below for variant behaviour. When a command terminates, an end-of-data is always sent to the output path (whether or not there has actually been any output).
The popup-selection button at top right of the panel controls the run mode. 'Normal' means that the command will just run once each time it is started; if it is processing standard input, it will (if well behaved.) quit when it gets an end-of-data. If you would rather that end-of-data signals on the input stream did not get passed to the command (so that it will keep on running until stopped) set the selection to 'Ignore E-O-D'. If you want it to get restarted automatically each time it terminates, use 'Auto-Restart'. Response to the 'Stop' button is slightly different in this last mode: the first click of the button just switches the mode back to 'Normal', so that the command will complete its processing if possible; if you need to really shut it down, click it a second time. [This is an attempt to bypass an apparent bug in the posix layer that causes 'orphan threads' if a 'kill' occurs at the wrong time. Sorry -- haven't been able to home in any further so far...]
When you save a diagram to a configuration file, the state of the element is preserved in that file. The command and directory are recorded (as text strings), as is the mode setting and whether or not the command is actually running when the save is performed.
The current version of the module only handles standard input and standard output. It does not, for example, directly handle standard error output. If you want to see this on the Output connector of the element, just redirect with the '2>&1' bash convention. Or of course you can redirect it elsewhere to a file or a named pipe.
As there is only one input stream, you cannot easily use commands such as 'diff' that by nature need two inputs, except again by using files or named pipes [which sort of seems to negate the 'visual pipeline' concept!]. More flexible possibilities are...possibilities... later.
Warning: There is a bug in the BeOS 'posix layer' when running on a PowerPC (though not on Intel systems) that can prevent multiple pipes from closing properly. This means that a program running under PipeStream may not close properly when an 'End-Of-Data' is received, and so an E-O-D may not get passed on. The problem is expected to be fixed in the next (4.1) release of BeOS.
This is also a multiconnector module, so that the input stream can be passed through to a succeeding element.
The two buttons at the top of the window should be fairly obvious. The left hand one will send any selected text; the other sends an 'End-Of-Data' signal, which will terminate the command in any PipeStream that receives it unless that is set to ignore these. If a StreamView receives an End-Of-Data, it displays a separator line (but takes no other action).
The StreamView window behaves like a standard BTextView, except in one respect: in 'Lines' mode the 'Enter' key only inserts a newline in the window when it is at the very end of the text, but it always sends the entire line. This lets you easily resend any line by moving the cursor anywhere within it and typing 'Enter', without disturbing the displayed layout. 'Ctrl-Enter' has essentially the reverse effect: it always inserts a newline, but never transmits the line. [Yes -- I borrowed this from Eddie...] This non-standard behaviour is only seem in 'Lines' mode (see below). (The display text is not saved, by the way, when you save the diagram.}
The 'Send as:' selector at top right of the panel determines which user actions result in data being transmitted. In the default state('Lines'), typed-in data is sent only when 'Enter' is typed, as a complete line. In 'Keystrokes' mode it is sent a character at a time, keystroke by keystroke. (Keystroke mode is only effective on typed-in data -- it doesn't change how a selected block is sent.) When the mode is 'Selected Only', nothing is ever sent as characters are entered or a new line is started: you must highlight the data you want to send and click 'SEND Selected'. In these latter two modes, the panel accepts input in a perfectly standard way -- 'Enter' is always echoed in the window. Also remember, though, that in Keystroke mode everything is sent, including things like cursor movement and deletions.
For convenience, the File Selector Panel remains visible once it is opened, so you can quickly transmit several files in sequence. (Note that the restriction on earlier versions that only one file could be selected at a time has been removed.) If you want to send multiple files in a particular order, you should drag (or select) them in that order separately, rather than as a group, as the order in which a group of files is processed is up to the Tracker's fancy alone.
The current directory in the filepanel is recorded in the configuration file when you save, but the filename itself is not. (This is new in this release.)
While a file is open to accept data, its name is displayed in the control panel. It will remain open until an End-Of-Data is received. If no file is open, the panel displays '<Closed>'. The current directory in the filepanel, though not any filename, is recorded in the configuration file when you save. (This is new in this release.)
Initially, when first placed, a SerialPort element is not attached to any port. Select the desired port from the popup menu of the top button ('Select') in its panel. Only one instance of the module can be attached to each available port.
If you need to alter any port settings from the defaults, use the menu brought up by the lower button ('Settings...'). Note that the 'Flow Control' submenu is disabled while a port is actually selected, because this mode cannot be changed on an active port; if you must change it, switch to 'NO CONN' in the port-selection menu, adjust the flow control mode, then reselect the desired port.
The current port, and all the settings, are saved when you make a configuration file, and will be restored when the file is invoked.
[Caution 1: On the BeBox, you will see -- in addition to 'serial1' through 'serial4' -- 'com3' and 'com4'. Don't select these latter two! They'll just hang the machine. I really don't know why they're there...]
[Caution 2: To send commands to a modem, you are likely to need to terminate them with the return character, rather than the newline generated by the 'Enter' key. This can cause endless confusion and annoyance, but I'm afraid you're on your own there.]
The 'Switcher' module is of limited use with the StreamWeaver, as it is really intended for reconfiguring paths in response to real-time 'director' messages. Because of the way in which most shell-based programs buffer their output, you don't see the data as it is generated (perhaps not until the stream is closed), so you can't respond as you might want to. No module to generate director messages would be of much use in the StreamWeaver, so none is included here. Look to the MusicWeaver for more significantly useful applications of the facility. The Switcher, however, can be controlled directly from its panel, so it can be useful to build diagrams that can be reconfigured with a mouse click.
Use it when, for example, you want both a StreamView and a ReadFile feeding the same PipeStream element.
Switcher is unique (here) in having three possible outputs, as there are two possible paths to which incoming data can be directed, and there is a second 'Control' input, with corresponding (pass-through) output. These latter connectors are for 'director' messages, and so not functional here.
When you open up a Switcher's panel, you will see a fairly large set of controls. At the top is a button/indicator, which is the only control of interest to us here. The large array of checkboxes below would control its response to 'director' messages. You can ignore this area. (You may want to just rezize the window so that only the button is visible.)
The indicator should initially be showing 'Pass'. This means that any data arriving at the input to the Switcher (the 'Through' connector) will simply be passed on out the corresponding 'Through' output. Pressing a mouse button over this control brings up a menu of the possible alternatives. Setting to 'Divert' will cause arriving data to be sent out the 'Divert' output rather than 'Through'. 'Dupe' means that all data will appear at both outputs (similar to a simple branch in the diagram). 'Stop' means that all data ends its journey here -- neither output sees it.
The state of a Switcher at the time a configuration file is generated is preserved there.
If you also have the MusicWeaver on your system, both this and the StreamWeaver should share the same top-level 'Weaver' directory, main Weaver program, and common modules. They have been designed to live together, and having more than one copy of the common stuff will probably just cause confusion.
It is now "AppreciationWare"... (:-)) You may use it as long as you like for any purpose, for no charge, but the author would appreciate any and all feedback! If you think it is worth five or ten bucks this would also be appreciated, but feel under no obligation. Please do feel obligated, however, to report uses, uselessness, gripes, feature requests, and the like. Thank you.
You are permitted to distribute the Weaver program, the StreamWeaver modules, and documentation in the form of the supplied ZIP archive without charge for non-commercial purposes, provided that the whole package is kept intact. For commercial use or distribution (other than charging a reasonable media and copying fee) please contact the author.
Pete Goodeve Berkeley, California e-mail: pete@jwgibbs.cchem.Berkeley.EDU pete.goodeve@computer.org