Visual Pipe-Fu
Columns
Tools Reference
Body
The body tool is used to truncate
a stream. You can eliminate the
head, tail or both from a given
text stream using offsets of line
numbers and/or identifying
strings.
a stream. You can eliminate the
head, tail or both from a given
text stream using offsets of line
numbers and/or identifying
strings.
With 'Torso' selected as the body part, the body tool will send everything in between the first
occurrence of Start Text + Offset up to Finish Text + Offset. You can use negative numbers for torso
as the offset and an offset of 0 will include the line containing the Start/Finish text. Select Head to
return all lines from the beginning of the stream up to the Finish offset. In the same vein, Tail returns
all lines from the end of the file to the start offset; using a negative number with head or tail will
return all but the specified number of lines, that is head with a -1 offset removes only the last line.
occurrence of Start Text + Offset up to Finish Text + Offset. You can use negative numbers for torso
as the offset and an offset of 0 will include the line containing the Start/Finish text. Select Head to
return all lines from the beginning of the stream up to the Finish offset. In the same vein, Tail returns
all lines from the end of the file to the start offset; using a negative number with head or tail will
return all but the specified number of lines, that is head with a -1 offset removes only the last line.
The Columns tool allows you to
select text out of rows by fields.
A delimited Column Type will
separate fields by the field
delimiter parameter set in the All
Tool section.
select text out of rows by fields.
A delimited Column Type will
separate fields by the field
delimiter parameter set in the All
Tool section.
The outgoing columns, marked by: Col 1, Col 2...., will contain the text that corresponds to the number
of the incoming column starting at 1 as entered into the 'Column' row. The 'Left' and 'Right' rows hold
the text that will appear to the left and right of the column value. If for instance you want to use a
comma as a field delimiter put a ',' in the Right field. If you want to use a TAB, use \t. You can specify
ranges in the form [start]-[finish] such as 2-7 which grabs the second through the seventh columns
or 1-last or 3-penult which grabs columns 3 through the penultimate or next-to-last.
of the incoming column starting at 1 as entered into the 'Column' row. The 'Left' and 'Right' rows hold
the text that will appear to the left and right of the column value. If for instance you want to use a
comma as a field delimiter put a ',' in the Right field. If you want to use a TAB, use \t. You can specify
ranges in the form [start]-[finish] such as 2-7 which grabs the second through the seventh columns
or 1-last or 3-penult which grabs columns 3 through the penultimate or next-to-last.
For Fixed Column types a field is
determined by a fixed number of
characters denoted by 'From'
and 'To'. The From offset begins
at zero and the To offset is
inclusive.
determined by a fixed number of
characters denoted by 'From'
and 'To'. The From offset begins
at zero and the To offset is
inclusive.
That means for the following input line:
The ants go marching two by two
a 'From' value of 0 and a 'To' value of 3 will return 'The'. Values of 1 and 3 will return 'he'. Remember
that a TAB character will count as one character, not the number of spaces that pad the field.
The ants go marching two by two
a 'From' value of 0 and a 'To' value of 3 will return 'The'. Values of 1 and 3 will return 'he'. Remember
that a TAB character will count as one character, not the number of spaces that pad the field.
Remember when using this tool
that if you are accessing a web
page you must put http:// in front of
the address to identify the domain.
that if you are accessing a web
page you must put http:// in front of
the address to identify the domain.
Scrape
Sort
When sorting a stream, each line
is treated as a record with fields
denoted by the field delimiter in
the All Tool section. To set a key
simply enter the column numbers
in the Key list.
is treated as a record with fields
denoted by the field delimiter in
the All Tool section. To set a key
simply enter the column numbers
in the Key list.
Records will be sorted alphabetically by key unless an 'N' is added to the field number to indicate a
numeric sort for that field. For example 1500 would sort with a lower value than 87 because 1 is
alphabetically lower than 8 unless you use the 'N' option. Setting the sort order to descending does
not reverse the priority of the keys; Key 1 will still be the primary key, Key 2 secondary and so forth. If
'Unique Keys' is checked, duplicates of any record with the same key will be eliminated even if the
records themselves differ.
numeric sort for that field. For example 1500 would sort with a lower value than 87 because 1 is
alphabetically lower than 8 unless you use the 'N' option. Setting the sort order to descending does
not reverse the priority of the keys; Key 1 will still be the primary key, Key 2 secondary and so forth. If
'Unique Keys' is checked, duplicates of any record with the same key will be eliminated even if the
records themselves differ.
Shell Execute
Shell Execute runs a program or a
script, sending the tool's incoming
text to the command's standard
input. The command's standard
output is captured as the tool's
outgoing text. If the shell
command is expecting Unix-style
newlines, check the Unix Newlines
box.
script, sending the tool's incoming
text to the command's standard
input. The command's standard
output is captured as the tool's
outgoing text. If the shell
command is expecting Unix-style
newlines, check the Unix Newlines
box.
This tool is disabled by default to make the user understand that special attention be given to ensure
that any third party scripts run only user-approved programs and scripts. To enable the tool simply
select 'Enable Shell Execute' from the 'Options' menu.
that any third party scripts run only user-approved programs and scripts. To enable the tool simply
select 'Enable Shell Execute' from the 'Options' menu.
Match
Match is similar to the grep utility
in unix. Use this tool to find
occurrences of strings or regular
expressions.
in unix. Use this tool to find
occurrences of strings or regular
expressions.
The regular expression library used is based on the Advanced Regex Engine developed by Henry
Spencer. An overview of its particulars can be found here.
Spencer. An overview of its particulars can be found here.
Translate
Translate turns occurrences of
one string or pattern into a given
string. Use of regular
expressions is the same as in
Match above.
one string or pattern into a given
string. Use of regular
expressions is the same as in
Match above.
This just saves the outgoing text
stream as a file. Make sure the
path is valid and writable or an
error will occur.
stream as a file. Make sure the
path is valid and writable or an
error will occur.
Save File
If you want the file to have Unix-style newlines, \n instead of \r\n, check the Unix Newlines box.
| Download Now |
Blackberry Extractor
To extract data as text from
your Blackberry, you first
create an ipd file with the
desktop software from
Research in Motion, which
can be downloaded from
here.
your Blackberry, you first
create an ipd file with the
desktop software from
Research in Motion, which
can be downloaded from
here.
Select a file for extraction by browsing or pick from a list by using the search function to find all ipd
files in a directory or on a disk drive. Select the database you wish to extract or 'Other' if it is not
listed. Every database in the ipd is available under 'Other'. Separator denotes an intra-field
delimeter, that is if multiple names are listed in the same Name field they will be separated by
'Separator'. Easy out will bring up a spreadsheet style grid that can be saved as a comma separated
value file (for import to Excel, Outlook, Lotus etc) while the Time Zone field is the number of hours
offset from Greenwich Mean Time. It is automatically filled with your computer's local value at startup
and the example of -6 is US Central Standard Time.
files in a directory or on a disk drive. Select the database you wish to extract or 'Other' if it is not
listed. Every database in the ipd is available under 'Other'. Separator denotes an intra-field
delimeter, that is if multiple names are listed in the same Name field they will be separated by
'Separator'. Easy out will bring up a spreadsheet style grid that can be saved as a comma separated
value file (for import to Excel, Outlook, Lotus etc) while the Time Zone field is the number of hours
offset from Greenwich Mean Time. It is automatically filled with your computer's local value at startup
and the example of -6 is US Central Standard Time.
ODBC Database Tool
Spreadsheet Tool
Data Source and User are pretty self-explanatory.
A word about Passwords. Passwords appear as dots in the application and are encrypted when
scripts are saved. The encryption binds the password to the script's filename so that if the script is
moved, the file must be re-saved after the password is re-entered, otherwise the password will not
be valid. Although encryption adds a measure of safety against data theft it must be noted, any
encryption algorithm can be broken. Security ultimately rests with users and system administrators.
SQL statements are executed as entered and can contain embedded replacement values. For
instance:
INSERT INTO teams
VALUES ("%3%", "%1%", "%2%")
will generate an insert statement substituting the third, first and second fields in a given row for %3%,
%1% and %2% respectively. To generate one statement per line check the 'By Line' box. Left
unchecked only one statement will be generated and executed. To test and review SQL statement
generation without submitting it to the database server, uncheck the 'Execute' box.
This tool makes use of an open source library available here, while our modified version is available
here.
A word about Passwords. Passwords appear as dots in the application and are encrypted when
scripts are saved. The encryption binds the password to the script's filename so that if the script is
moved, the file must be re-saved after the password is re-entered, otherwise the password will not
be valid. Although encryption adds a measure of safety against data theft it must be noted, any
encryption algorithm can be broken. Security ultimately rests with users and system administrators.
SQL statements are executed as entered and can contain embedded replacement values. For
instance:
INSERT INTO teams
VALUES ("%3%", "%1%", "%2%")
will generate an insert statement substituting the third, first and second fields in a given row for %3%,
%1% and %2% respectively. To generate one statement per line check the 'By Line' box. Left
unchecked only one statement will be generated and executed. To test and review SQL statement
generation without submitting it to the database server, uncheck the 'Execute' box.
This tool makes use of an open source library available here, while our modified version is available
here.
Select a spreadsheet file for
extraction/insertion by clicking
Browse or search directories
recursively with Search. Sheet
specifies the sheet name and
Coordinates indicates the
insertion/extraction point.
extraction/insertion by clicking
Browse or search directories
recursively with Search. Sheet
specifies the sheet name and
Coordinates indicates the
insertion/extraction point.
To extract data as text from
your Blackberry, you first
create an ipd file with the
desktop software from
Research in Motion, which
can be downloaded from
here.
your Blackberry, you first
create an ipd file with the
desktop software from
Research in Motion, which
can be downloaded from
here.
Coordinates take the Excel form of LetterNumber like C5 to reference column 3, row 5. Insert takes
a single such coordinate and will insert text as rows with each field (determined by 'Field Delimiter')
occupying a cell. Extraction requires a rectangle with two coordinates separated by a colon, such
as: B2:M8.
Special thanks to Yap Chun Wei for his BasicExcel class library available here. Our modified
version is available here.
a single such coordinate and will insert text as rows with each field (determined by 'Field Delimiter')
occupying a cell. Extraction requires a rectangle with two coordinates separated by a colon, such
as: B2:M8.
Special thanks to Yap Chun Wei for his BasicExcel class library available here. Our modified
version is available here.
Clipboard Tool
There's not much on this panel because there's not much to the clipboard.
Direction indicates whether text is to be copied or pasted. This tool will not
work when run from the command line since that is a DOS program and the
clipboard is a Windows facility.
Direction indicates whether text is to be copied or pasted. This tool will not
work when run from the command line since that is a DOS program and the
clipboard is a Windows facility.
To use the ODBC database tool
you must first setup an ODBC
datasource. It's easy to do, but if
you haven't done it before check
out the tutorial here.
you must first setup an ODBC
datasource. It's easy to do, but if
you haven't done it before check
out the tutorial here.
