NAME
mk_dmc_files - generate network or single station dataless or VND file with comment on changes
SYNOPSIS
mk_dmc_files
[-v]
[-z]
[-C]
[-m]
[-p pf]
[-d output_dir]
[-N net]
[-s sta]
[-f output_file]
[-o orb]
{-D | -V vnet}
dbin dbtrack [comment]
SUPPORT
Contributed code: NO BRTT support.
THIS PIECE OF SOFTWARE WAS CONTRIBUTED BY THE ANTELOPE USER COMMUNITY. BRTT DISCLAIMS ALL OWNERSHIP, LIABILITY, AND SUPPORT FOR THIS PIECE OF SOFTWARE.
FOR HELP WITH THIS PIECE OF SOFTWARE, PLEASE CONTACT THE CONTRIBUTING AUTHOR.
DESCRIPTION
mk_dmc_files is designed to create and track the updates a network
operator makes to a dataless SEED volume, virtual network definition (VND).
When changes were made and what they were is important
for dynamic networks such as the USArray Transportable Array.
This script generates either (1) a single station or combined network dataless
SEED volume or (2) a virtual network definition file. It allows for a comment on
why the update was made to be tracked in an extension table, dmcfiles.
The output table can then be queried to retrieve the comment for products
such as webpages. Of course, the results of such a query can only be as good
as the comments that are entered. The output dmcfiles extension table
is also used by the ims2dmc program.
mk_dataless_seed is called to generate the dataless volume. deployment2vnd
is called to generate a Virtual Network Definition file.
OPTIONS
-
-v
Verbose output.
-
-z
Use gzip to compress the output dataless SEED volume.
-
-C
Use a product directory underneath which vnc_dir, sta_dataless_dir or dataless_dir are created. The
default is to create the vnd_dir, sta_dataless_dir and dataless_dir in the current directory.
This is very useful when used in conjunction with the -o option. With -o,
the file to be transferred is stored in the product_dir/vnc_dir product_dir/dataless_sta or
product_dir/dataless_dir area, but the file transferred via orbxfer appears as
vnd_dir/YYYY/VNET_YYYYMMMDD.csv/ on the remote server. Ignored if -d is used. The directory to be
used is specified in the mk_dmc_files.pf parameter file. Previous versions of this script made you
specify the directory on the command line for each run of the program. See EXAMPLES.
-
-m
Generate an automated email informing the mail_list of an updated file. The filename,
comment, and orb (if -o) are included in the text of the email. The recipient list is specified in the
mk_dmc_files.pf parameter file. Previous versions of this script made you specify the recipients
on the command line for each run of the program.
-
-p parameter_file
Parameter file to use. The default parameter file name is mk_dmc_files.
-
-d output_dir
Output directory for dataless SEED volume or vnd file. This overrides the default
directory, dataless_dir , sta_dataless_dir, or vnd_dir, in the
parameter file and no year or station directory is created. Cannot use with -C.
-
-N net
Network code to use in the output dataless SEED name. This overrides the default
net code from the site.pf parameter file. Irrelevant if -V option is selected.
-
-s sta
Generate a dataless seed for this single station. Single station dataless are stored
in the sta_dataless_dir directory. Irrelevant if -V option is selected.
-
-f filename
Output dataless SEED or VND filename. The default filename for a dataless SEED
volume is: DATALESS.XX.YYYY.MM.DD or DATALESS.XX_STA.YYYY.MM.DD where
XX is replaced with the value of -N, the default_seed_network from site.pf,
or is left as XX if these other two values are undefined. The default filename
for a VND is: VNET_YYYYMMDD.csv where VNET is replaced with the value given
to -V. If the -z option is used the output file name has .gz appended to it. See
BUGS AND CAVEATS below if using -z with the -o option.
-
-o orb
Output orb for orbxfer2(1). Files are placed into the orbxfer2 server orb
using the vnd_dir and dataless_dir from the pf file. Because of this,
you might not want to use a path containing the filesystem root. Anyone on the receiving
side of orbxfer2 that is attempting to write the files out using the same directory
structure would have a long path to look down if a root level directory structure is used.
Note however that the absolute path is placed into the dir field in the output
dmcfiles table. Look at the -C option if you have a long path to where the
files are stored, but do not want to transfer that full path with orbxfer. Also
see BUGS AND CAVEATS below.
-
-V vnet
Generate a Virtual Network Definition (VND) file using the specified virtual network
code. These codes are created and maintained by the DMC. Example vnets include: _US-TA,
_US-EARN, _CASCADIA-TA, _US-BB, _OBSIP, _ANSS, etc. See http://www.iris.edu/vnets.
Either -V or -D must be specified on the command line.
-
-D
Generate a dataless SEED volume rather than a VND file. Either -V or -D must be specified
on the command line.
-
dbin
Input database from which the dataless or VND is generated. See mk_dataless_seed(1) or
deployment2vnd(1) for requirements. This argument is required.
-
dbtrack
Database where the tracking table, dmcfiles exists. This may be different
than dbin. This argument is required.
-
comment
A descriptive comment that describes what prompted you to make a new
dataless SEED volume or changes to the VND. The maximum string length is 180 characters. The
parameter file has a very generic default comment if you do not specify one
on the command line. You might consider attempting to use consistent comments for ease in
use of subsetting for web page or other scripts that interact with the dmcfiles
table. The maximum comment length is 180 characters.
ENVIRONMENT
Need to have sourced $ANTELOPE/setup.csh and environment variable
$PFPATH set.
PARAMETER FILE
Below is the default
mk_dmc_files parameter file.
# mk_dmc_files.pf
vnd_dir vnd # year based directories are created under here
dataless_dir dataless # station or year based directories are created under here
sta_dataless_dir dataless_sta # station based directories will be created under here
product_dir /anf/TA/products # path where data products are stored.
# i.e. ending location would be a station dataless volume is: $product_dir/$dataless_sta/$sta/
# Only used with -C option. If left blank, products appear in ./ under appropriate vnd|dataless|sta_statless directories
mailto someone@ucsd.edu,someone@iris.washington.edu # comma separated list of email recipients, overridden by command line -m
# comment must be a single line less than 180 characters
default_comment &Literal{
Metadata change
}
This is a very basic parameter file that specifies default output
directories and a basic comment.
-
vnc_dir
Output directory where VND files are saved. If using the orbxfer2
-o option, this is the base directory that the receiver catches
the files in. Do not use a directory starting at the root level if
the files are going to be transferred with orbxfer2. Use the -C option if
you wish to transfer files with orbxfer2 that will be received below the
vnc_dir, but reside under a different directory on your storage system.
-
dataless_dir
Output directory where whole network dataless files are saved.
-
sta_dataless_dir
Output directory where single station dataless files are saved.
-
product_dir
Top level directory where product files are saved. When the -C option is used, all
dataless_dir, vnd_dir, and sta_dataless_dir directories are created
under product_dir. The intent is that when the -o option is used, orbxfer(1)
will not include this top level directory with the file transfer.
-
mailto
Comma separated list of email addresses which receive notification of updated files.
-
default_comment
The default comment to be placed in the dmcfiles table. The
maximum length of this string is 180 characters.
EXAMPLE
Generate a dataless for station 109C and keep it in compressed
format after a datalogger swap. Do not transfer it via orbxfer2.
% mk_dmc_files -D -z -s 109C dbmaster/ta_only
dbops/usarray "109C datalogger swap"
% ls dataless_sta
109C/
% ls dataless_sta/109C/
DATALESS.TA_109C.2011.04.01
Generate a VND for the _US-TA virtual network after adding two new stations.
Store it in the local vnd_dir specified in the default parameter file. Transfer
it via orbxfer2.
% mk_dmc_files -V _US-TA -o anfexport:meta \
dbmaster/anf dbops/anf "Added X21A Z22A"
Generate a single station dataless, store it in a product directory
and inform people why the dataless was generated. The dataless will
end up in the directory: /raid/data/products/TA/dataless_sta/L08A and
will not be transferred via orbxfer2.
% mk_dmc_files -m -C -D -s L08A anfexport:meta \
dbmaster/anf dbops/anf "L08A station closed"
% pfecho mk_dmc_files
dataless_dir dataless_all
default_comment &Literal{
Metadata change
}
mailto jeakins@ucsd.edu,someone@iris.washington.edu
product_dir /raid/data/products
sta_dataless_dir dataless_sta
vnd_dir vnd
Generate a complete dataless for a network after adding two new stations.
Store it in a data products directory on your raid system, but transfer using orbxfer2
with only the dataless_dir appearing for the receiver.
% mk_dmc_files -D -C -o anfexport:meta
dbmaster/ta_only dbops/usarray "Added X21A Z22A"
In the above case the file generated would be something like
/raid/data/products/dataless/2008/DATALESS.TA.2008.03.25. It would
appear in the anfexport:meta orb as a source that looked like:
anfhost/xfer/76596/DATALESS.TA.2008.03.25
If a downstream user was connected to the anfexport:meta
orb with orbxfer2 in the receiver mode and chose to have the
directories preserverd, they would receive the file on their
host as:
dataless/2008/DATALESS.TA.2008.03.25
DIAGNOSTICS
Errors from the system calls to mk_dataless_seed(1) or deployment2vnd(1) are
not trapped properly by the elog routines.
Make sure your input database has no egregious errors.
SEE ALSO
mk_dataless_seed(1)
deployment2vnd(1)
orbxfer2(1)
dmcfiles(5)
db2ims(1)
ims2dmc(1)
BUGS AND CAVEATS
Earlier versions (prior to 5.1-64) required arguments after the -m and -C
options. Those arguments have been pushed into the parameter file.
This is an in-house script necessary for the functionality of the USArray
Transportable array. Please let me know if you find it useful, or seriously
lacking in features that might help your group's needs.
If there is no reader attached to the specified output orb with -o, the program
hangs until a reader is attached. To avoid this behavior, the wait_match
parameter in the orbxfer2.pf file must be blank.
The 4.9 and earlier versions of orbxfer2 had a bug such that compressed files
pushed into the orb would retain there file names (i.e. myfile.gz), but would
actually be uncompressed before placement in the orb. The receiver would then
get myfile.gz out of the orb, but it would be an uncompressed file. I believe
this bug has been resolved.
If there are permissions problems with the output orb used with -o, the transfer
of the file to the orb may fail silently. Review the orbserver logs to see if
there is a problem.
All of the different directory options are confusing. I included them rather
than choosing a single method that I commonly use. I would be interested to
hear of other ways people might use this program and modifications that might
be required or clarifications that you would find helpful.
See the current SEED manual for a full description of a dataless SEED volume.
Converse with the DMC regarding the format of the VND files.
AUTHOR
Jennifer Eakins
Univ. of California, San Diego
Antelope User Group Contributed Software