NAME
mk_dbops - update dbops tables after station install, removal, adoption, transition or comms change
SYNOPSIS
mk_dbops
{-I | -U | -R | -A }
[-k]
[-n]
[-v]
[-d on|off|string]
[-e AC|solar|wind|string]
[-V vnet]
[-m source_match]
[-p pf]
[-C cmdORB]
[-P prelimORB]
[-S statusORB]
[-w prelimwfDB]
[-W wfDB]
[-s siteDB]
dbops snet sta [timestamp [comm_provider [comm_type]]]
SUPPORT
Contributed code: NO BRTT support.
THIS PIECE OF SOFTWARE WAS CONTRIBUTED BY THE ANTELOPE USER COMMUNITY. BRTT DISCLAIMS ALL OWNERSHIP, LIABILITY, AND SUPPORT FOR THIS PIECE OF SOFTWARE.
FOR HELP WITH THIS PIECE OF SOFTWARE, PLEASE CONTACT THE CONTRIBUTING AUTHOR.
DESCRIPTION
Background
The EarthScope USArray Transportable Array (TA) network of 400+
currently operational seismic stations and approximately 1800 lifetime
installations, prompted development of additional relations (tables)
to keep track of operational information at the Array Network Facility (ANF).
Tables were created to keep track of communication setup, inclusion time
in the virtual network, and adoption status if the station is still
operational. Many tools have been developed to make use of this information.
See for instance,
webdlmon(3H),
deployment2vnd(1),
rt_baler_info(1),
rt_daily_return(1), and
orbmonrtd_pf_generator(1). For the first 700
or so stations, the deployment and comm tables were edited using
dbe(1):
a slow, tedious process that was prone to error.
Dataflow and Procedures at the ANF
At the ANF, newly installed stations are quarantined into a prelim ORB until
metadata is available. Data is also written out to disk into a prelim DB as
a precaution against a corruption of the prelim ORB, or the overwrite of the
first prelim data received.
Upon receipt of the metadata, dbbuild(1) is used to update the dbmaster
tables. A dataless SEED file is generated and passed to downstream users via
mk_dmc_files(1) (which uses orbxfer2(1)). The quarantined data is
now transferred via orb2orb(1) to the certified ORB where it is written
out to disk with orb2db(1).
At this point, following the old procedures, the deployment, comm,
and dlsite tables would be updated via dbe(1) in a process which
took significant time and concentration and led to various errors due to typos
or omission. As the amount of stations increased, a new method for updates was
called for and the development of mk_dbops began.
New approach
In order to reduce errors and to make the process easier and less time consuming,
mk_dbops was written to automatically update the adoption, comm, deployment, and
dlsite tables when there is: a new station installation; a station removal or adoption;
or a communications update to an existing station. Instead of having
to fill the fields by hand, the various times are collected from existing databases
or orbs whenever possible. A parameter file maintains the default locations of
these orbs and dbs, with command line overrides possible.
OPTIONS
See the EXAMPLE section for how needed command line options differ depending on the type
of update (installation, comms update, removal, or adoption). One and only one of -I,
-U, -R or -A must be used.
-
-I
Station installation. Updates occur in three tables: comm, deployment,
and dlsite. Requires that timestamp be the station certification time.
-
-U
Communication update for pre-existing station. Updates occur in
two tables: comm and dlsite. Requires that timestamp be the time of new
communications. End time for previous communication will be 1 second earlier
than timestamp.
-
-R
Station removal. Updates occur in three tables: comm, deployment,
and dlsite. Requires that timestamp be the station decertification time.
If the station operation is taken over by another group, use the
-A option instead.
-
-A
Station adoption. Updates occur in adoption and deployment tables. Updates may
also occur in dlsite and comm if the adoption type is not EARN. This option starts
an interactive mode where additional information is requested. When prompted fill
in: the adoption type, new vnet code if it is an EARN or TRANSITION adoption type,
the new station name; new network code that the station will be
operating under; responsible data center (15 char max).
If you do not have information on the new station name, the sta and snet
from the command line is used (defaults shown after prompt). When there is a
station adoption, the deployment.equip_remove is left null.
Currently there are only four adoption_types available: E (EARN), T (OPERATOR TRANSITION),
R (REFNET/BACKBONE), and X (CEUSN/1N4). See PARAMETER FILE section below for additional description.
-
-k
Keep created backup files. Backups of the tables to be modified are always created
and appear as dbops.table+. Any backup files already existing
are overwritten. This option overrides the default behavior which is to remove
these backup files upon successful completion of the script.
-
-d duty_cycle_status
Text indicating whether or not the comms are on a duty cycle. Typical values are
yes or no. Default is the null value "-".
-
-e power_type
Text indicating the type of power used at the station. Typical values are
AC, solar, wind or unknown. Default is the null value "-".
-
-V vnet
Virtual network code to be used in the deployment table. Overrides default
found in pf. Only valid when using -I, an installation or when the deployment
table has multiple records for a single station (i.e. both a _US-TA and _US-EARN record).
-
-P prelimORB
Preliminary ORB used to gather the deployment.time value during an installation, -I.
Overrides the default found in pf.
-
-w prelimwfDB
Use the preliminary waveform database during an installation, -I, to
gather the deployment.time value rather than the value obtained from
the prelimORB.
Use the waveform database during a removal, -R, to gather the
deployment.endtime value. Overrides the default found in pf.
-
-m source_match
Packet source match used during installation update, -I, for collecting
the deployment.time value. The default looks for packets which match snet_sta.*M40.
-
-S statusORB
Status ORB used by the q330_location(1) program during an installation, -I if -n
is not selected. Overrides the default found in pf.
-
-n
Do not run the q330_location(1) program during an installation, -I. You will
need to update the dlsite table outside of this program (using q330_location(1) or by
other means). The q330_location(1) program has verbose output and is limited to the
station you are attempting to update. See q330_location(1).
-
-W wfDB
Use this certified waveform database during a removal, -R, to gather the
deployment.endtime value. Overrides the default found in pf.
-
-s siteDB
Database containing stage table if different from dbops database. stage.time
is used for comm.time during an installation, -I and for deployment table
updates for an adoption, -A. stage.endtime is used
for comm.endtime and dlsite.endtime during a removal, -R.
-
-p pf
Parameter file to use. Default pf is mk_dbops.pf.
-
-v
Verbose log output.
-
dbopsdb
Database containing adoption, comm, deployment, and dlsite tables that are to be updated.
Be sure to use -s if dbopsdb does not also contain the stage table.
-
snet
Seed network code for station. Note this is different from the virtual network code.
-
sta
Station code for station to be installed/updated/removed/adopted. The snet and sta are
combined to make a dlname used in the dlsite table.
-
timestamp
Must be a valid timestamp. If you use a date string, surround timestamp by quotes (i.e. "10/28/2008 18:30:00").
For an installation, this is the certification time used in the deployment table.
For an update, this is the time that the communications change occurred.
For a removal, this is the decertification time used in the deployment table.
For an adoption, this is the decertification and certification time used in the deployment table.
-
comm_provider
String describing who provides the communications (i.e. Wild Blue, ATT, Local TelCo, etc.). This
is mandatory for an installation and an update. It is ignored for removals and adoptions. If you have
spaces in the provider name, surround comm_provider by quotes (see EXAMPLES). This value is
checked against the accepted_comm_providers found in the parameter file. A non-fatal alert is
reported if the comm_provider supplied is not found in accepted_comm_providers.
-
comm_type
String describing the type of communications (i.e. vsat, dsl, cell modem, etc.). This
is mandatory for an installation but not for an update. It is ignored for removals and
adoptions. If it is not included during an update, the pre-existing comm.commtype is used.
If you have spaces in the communications type, surround comm_type by quotes (see EXAMPLES).
This value is checked against the accepted_comm_types found in the parameter file.
A non-fatal alert is reported if the comm_type supplied is not found in accepted_comm_types.
FILES
There are four schema extension tables that are modified by this program: comm, deployment,
dlsite, and adoption. See the documentation available via dbhelp(1) for the css3.0
schema. See also the additional documentation available in deployment(5), adoption(5),
and comm(5).
ENVIRONMENT
Needs to be run under an ANTELOPE environment.
PARAMETER FILE
Here is an example pf file:
#
status_orb somewhere.ucsd.edu:status # orb where status packets exist, override with -S
prelim_orb somewhere.ucsd.edu:prelim # orb where prelim waveforms exist, override with -P
cmd_orb somewhere.ucsd.edu:prelim # orb where dlcmds are sent, override with -C
wfdb db/usarray # where certified waveforms are written to disk, override with -W
vnet _US-TA
pdcc IRIS DMC
packet_match .*M40|.*M100 # packet match for those in the prelim orb
channel_match HNZ|[BH]HZ # channels to check wfdisc start/end times and open records in stage table
# Only 4 adoption types supported: E, T, R, X.
# Do not change or add to list of codes below.
# atype_phrase can be modified
adoption_types &Arr{
# code atype_phrase
E EARN
T OPERATOR TRANSITION
R REFNET / BACKBONE
X CEUSN/1N4
}
accepted_comm_types &Tbl{
DSL
WiFi
cable
cell modem
no comms
orb
orb2orb
regular internet
slink2orb
vsat
}
accepted_comm_providers &Tbl{
ATT
Cal_Ore tel
DSL Frontier
DoE
etc.
}
This parameter file maintains the default locations for various orbs and dbs
such that they do not have to be repeated on the command line unless overrides
are required.
-
status_orb
Status ORB used by the q330_location program during an installation, -I if -n
is not selected. If the orb is not operational or you do not have access, the program exits.
-
prelim_orb
Preliminary ORB used to gather the deployment.time value during an installation, -I.
If the orb is not operational or you do not have access, the program exits.
-
cmd_orb
Command ORB used to send and receive dlcmds. Used only during an installation (i.e.
when the -I option is selected. Ignored if the -n is used.
If the orb is not operational or you do not have access, the program exits.
-
wfdb
Use this certified waveform database during a removal, -R, to gather the
deployment.endtime value.
-
vnet
Virtual network code to be used in the deployment table.
-
pdcc
Data center to be filled in for deployment.pdcc. If not filled in, the pdcc is set to -.
The DMC has some limitation on what can be put in this field. Contact them for a full definition of a PDCC.
-
packet_match
Match statement for packets in the preliminary orb. These packets are used for start times when using the -I option
for new installations. The orbselect statement will be snet_sta_(packet_match)
-
channel_match
Channel subset to use for checking start/end times in wfdisc for installations and removals. The stage table is also subset
to review for equipment install times using this subset. Ideally, this would be one vertical channel. If you have multiple
sensors with different channel codes installed at different times, the earliest start time in the stage table for the given
station and channel_match will be used.
-
adoption_types
Only three adoption types are accepted E, T, and R. These correspond to an EARN transition,
a transition to regional/national lab network operator, and a transition to operation as a REFNET
or Backbone station. The atype_phrase is placed in the adoption table. The phrase can be modified,
but a standard should be chosen for compatibility with web apps that may use the adoption.atype
field for sorting.
-
accepted_comm_types
A list of comm types that one would expect to see in the comm table. If you use one that is not listed here
the script alerts but does not die.
-
accepted_comm_providers
A list of comm providers that one would expect to see in the comm table. If you use one that is not listed here
the script alerts but does not die.
EXAMPLE
There are four different modes of operation for this program: installation,
update, removal, or adoption.
For a new installation:
mk_dbops -I [-k] [-v] [-n] [-V vnet] [-m source_match] [-p pf] [-C cmdORB] [-P prelimORB] [-S statusORB] [-w prelimwfDB] [-s siteDB] dbopsdb snet sta certify_time comm_provider comm_type
For a comms update:
mk_dbops -U [-k] [-v] [-p pf] dbopsdb snet sta time_of_comm_change comm_provider [comm_type] ;
For a station removal:
mk_dbops -R [-k] [-v] [-p pf] [-W wfDB] [-s siteDB] dbopsdb snet sta decert_time
For a station transition to regional network:
mk_dbops -A [-k] [-v] [-p pf] [-W wfDB] [-s siteDB] dbopsdb snet sta decert_time
Installation
Update the comm, deployment, and dlsite table after installation of
TA TEST a vsat site with comms provided by Wild Blue. Certification time
is 12/10/2008 18:30 UTC.
/B>
% mk_dbops -I dbops/usarray TA TEST \
"12/10/2008 18:30:00" "Wild Blue" vsat
/B>
Update the comm, deployment, and dlsite table after installation of
BB TST2 a cell modem site with comms provided by Alltel and part of the _US-TEST
vitural network. Certification time is 12/10/2008 18:30 UTC. Use the
prelim waveform database for the first data available time, deployment.time.
% mk_dbops -I dbops/usarray -V _US-TEST -w prelim/usarray \
BB TST2 "12/10/2008 18:30:00" "Alltel" "cell modem"
Update
Update the comm and dlsite tables after the communications
provider for TA EXST changed from Verizon to ATT. Time new comms were
established is 12/10/2008 18:30:00. Keep backups of the comm and dlsite tables.
% mk_dbops -k -U dbops/usarray TA EXST \
"12/10/2008 18:30:00" ATT
Removal
Update the comm, deployment and dlsite tables for TA DONE. Decertification
(time metadata was passed along closing the station) was 12/10/2008 18:30:00.
Keep backups of the comm, deployment and site tables.
% mk_dbops -k -R dbops/usarray TA DONE 2008:353:18:30:00
Adoption/Transition
Update the adoption, comm, deployment and dlsite tables, after
station TA ADPT is adopted by PNSN. Time station transitioned
to non-standard TA opertations was 12/10/2008 18:30:00. Station
was adopted by UW and called SNOW.
% mk_dbops -A dbops/usarray TA ADPT "12/10/2008 18:30:00"
mk_dbops: Adoption type(E|T|R|X|-): T
mk_dbops: New station name(ADPT): SNOW
mk_dbops: New network code(TA): UW
mk_dbops: Newly responsible data center(????????): PNSN
Update the comm, deployment and dlsite tables, after transition of
station TA ERNN. Time station transitioned to non-standard TA operations
was 12/10/2008 18:30:00. Station is part of the EARN program and will continue
to flow through the ANF with the same name.
% mk_dbops -A dbops/usarray TA ERNN "12/10/2008 18:30:00"
mk_dbops: Adoption type(E|T|R|-): E
mk_dbops: New vnet code for EARN station(_XX-XXXXX): _US-EARN
mk_dbops: New station name(EARN):
mk_dbops: New network code(TA):
mk_dbops: Newly responsible data center(-): ANF
Update the comm and deployment tables of the usarray database and the
comm and deployment table of the ceusn database, after transition of
station TA A01A. Time station transitioned to N4/CEUSN operations
was 03/13/2015 18:00:00.
% mk_dbops -n -A db/usarray TA A01A "03/13/2015 18:00:00"
mk_dbops: Adoption type (E|T|R|X|-): X
mk_dbops: New station name (A01A):
mk_dbops: New network code (TA): N4
mk_dbops: Newly responsible data center (????????): ANF
mk_dbops: New vnet code for CEUSN/1N4 station (_XX-XXXXXXXX): _CEUSN
mk_dbops: New database for deployment and comms (new/db): n4/dbops/ceusn
DIAGNOSTICS
-
No matching records in stage table for stations XXXX
This is a common message that you will run across if your dbmaster tables are not
updated for the type of dbops change you are trying to make. There are multiple
things to check:
-
Are you matching the proper channels available for that station?
The parameter file has a channel sifter in it called chan_match. For instance,
if you have BHZ channels in your metadata and the parameter file is set to use HHZ channels,
you will see this error.
-
Have you closed off the records for that station in the metadata?
If you are working with a removal, -R, or an adoption, -A with an
adoption type of T|transition or X|CEUSN/N4 conversion you need to have updated
the dbmaster tables to indicate the station is closed.
-
Have you added the metadata for that station to the database?
If you are working on an installation, -I, the metadata for that station
must be in the input database before running mk_dbops.
-
couldn't find ip address for 'somewhere.ucsd.edu' : 'Name or service not known'
This means you have no updated your mk_dbops.pf file to point to the appropriate orbs
(status, prelim or cmd orbs) or you have not used the command line overrides -S, -P,
or -C.
-
Can't select XX_XXXX(.*M40|.*M100) packets from prelim orb, somewhere.ucsd.edu:prelim
Perhaps you are not running a prelim orb to segregate the data. Check that you have your orbs
specified correctly.
-
No records in certified wf db matching N25K for chan=~/HNZ|[BH]HZ/
Check the values of wfdb or possibly channel_match in mk_dbops.pf. You likely are
pointing at the wrong database where the waveforms reside. Alternatively, if you are using the -W
option, you could be pointing at the wrong database.
SEE ALSO
adoption(5)
comm(5)
deployment(5)
convert_comm(1)
convert_dlsite(1)
convert_deployment(1)
deployment2vnd(1)
mk_dmc_files(1)
q330_location(1)
BUGS AND CAVEATS
This script is firmly founded upon assumptions made in the operational model
of your network. Testing has not been attempted for setups outside what is
in use at the ANF. For instance, you may run across errors if you do not
operate separate
prelim and
cmd orbs.
Per agreed upon conventions: EARN stations retain their same snet_sta,
TRANSITION stations may change their snet_sta, and REFNET/Backbone
stations should retain their same snet_sta. If these conventions change,
reprogramming, or at least further testing of the current script is necessary.
If using
-I mode with the default behavior to pull the oldest data time from
the prelim orb, you must have permission to access the orb.
If using
-I or
-U mode that checks the status orb,
you must have permission to access the orb.
User must have write access to dbops tables (adoption, comm, deployment, and dlsite).
If a record cannot be added to a table the program dies. This could be a problem,
for instance, if one table had a successful update (i.e. comm) and then the update
to the deployment table fails. Prior to any updating, the script creates backups
of all tables to be updated. They are called
dbopsdb.table+ and are removed
upon successful completion of the script unless
-k is specified.
When updating the dlsite table using
-I mode, the
q330_location code
is run. As of the 4.11 release, this
q330_location script subsets based on
a single station. Earlier releases did not subset based on a single station, so
all stations were reviewed and reported upon. This caused some problems with null
values for commtype and provider if there were multiple stations that had not been
added to the deployment table. Also note that
q330_location has copious output.
If dbmaster tables are not updated for a newly installed station,
you will not be able to update the tables in
-I mode.
When running in
-I mode, if the newly certified stations have no data
in the default
status_orb you will have to run
q330_location(1)
by hand. Alternatively, use the
-S option to point at the orb that has the
necessary .*/st packets, likely the
prelimORB.
When running in
-U mode, if you only have duty cycle status or power type to
update, you will still need to put in the comm provider in order to convince the script
to update the comm table.
The deployment table subset has been modified to include the vnet. This should
deal with instances for example where you had a station operational as TA, it was
adopted/transitioned to operations as N4, and then you have to close the station.
AUTHOR
Jennifer Eakins
ANF-IGPP-SIO-UCSD
jeakins@ucsd.edu
Antelope User Group Contributed Software