• Antelope Release 5.5 Mac OS X 10.8.5 2015-04-21

 

NAME

mk_dbops - update dbops tables after station install, removal, adoption, transition or comms change

SYNOPSIS

mk_dbops 
	{-I | -U  | -R | -A }
	[-k]
	[-n]
	[-v]
	[-d on|off|string]
	[-e AC|solar|wind|string]
	[-V vnet]
	[-m source_match]
	[-p pf]
	[-C cmdORB]
	[-P prelimORB]
	[-S statusORB]
	[-w prelimwfDB]
	[-W wfDB]
	[-s siteDB]
	dbops snet sta [timestamp [comm_provider [comm_type]]]

SUPPORT


Contributed code: NO BRTT support.
THIS PIECE OF SOFTWARE WAS CONTRIBUTED BY THE ANTELOPE USER COMMUNITY. BRTT DISCLAIMS ALL OWNERSHIP, LIABILITY, AND SUPPORT FOR THIS PIECE OF SOFTWARE.

FOR HELP WITH THIS PIECE OF SOFTWARE, PLEASE CONTACT THE CONTRIBUTING AUTHOR.

DESCRIPTION

Background

The EarthScope USArray Transportable Array (TA) network of 400+ currently operational seismic stations and approximately 1800 lifetime installations, prompted development of additional relations (tables) to keep track of operational information at the Array Network Facility (ANF). Tables were created to keep track of communication setup, inclusion time in the virtual network, and adoption status if the station is still operational. Many tools have been developed to make use of this information. See for instance, webdlmon(3H), deployment2vnd(1), rt_baler_info(1), rt_daily_return(1), and orbmonrtd_pf_generator(1). For the first 700 or so stations, the deployment and comm tables were edited using dbe(1): a slow, tedious process that was prone to error.

Dataflow and Procedures at the ANF

At the ANF, newly installed stations are quarantined into a prelim ORB until metadata is available. Data is also written out to disk into a prelim DB as a precaution against a corruption of the prelim ORB, or the overwrite of the first prelim data received.

Upon receipt of the metadata, dbbuild(1) is used to update the dbmaster tables. A dataless SEED file is generated and passed to downstream users via mk_dmc_files(1) (which uses orbxfer2(1)). The quarantined data is now transferred via orb2orb(1) to the certified ORB where it is written out to disk with orb2db(1).

At this point, following the old procedures, the deployment, comm, and dlsite tables would be updated via dbe(1) in a process which took significant time and concentration and led to various errors due to typos or omission. As the amount of stations increased, a new method for updates was called for and the development of mk_dbops began.

New approach

In order to reduce errors and to make the process easier and less time consuming, mk_dbops was written to automatically update the adoption, comm, deployment, and dlsite tables when there is: a new station installation; a station removal or adoption; or a communications update to an existing station. Instead of having to fill the fields by hand, the various times are collected from existing databases or orbs whenever possible. A parameter file maintains the default locations of these orbs and dbs, with command line overrides possible.

OPTIONS

See the EXAMPLE section for how needed command line options differ depending on the type of update (installation, comms update, removal, or adoption). One and only one of -I, -U, -R or -A must be used.

FILES

There are four schema extension tables that are modified by this program: comm, deployment, dlsite, and adoption. See the documentation available via dbhelp(1) for the css3.0 schema. See also the additional documentation available in deployment(5), adoption(5), and comm(5).

ENVIRONMENT

Needs to be run under an ANTELOPE environment.

PARAMETER FILE

Here is an example pf file:


#
status_orb      somewhere.ucsd.edu:status	# orb where status packets exist, override with -S

prelim_orb      somewhere.ucsd.edu:prelim	# orb where prelim waveforms exist, override with -P

cmd_orb         somewhere.ucsd.edu:prelim	# orb where dlcmds are sent, override with -C

wfdb            db/usarray	# where certified waveforms are written to disk, override with -W

vnet            _US-TA

pdcc            IRIS DMC

packet_match    .*M40|.*M100                    # packet match for those in the prelim orb

channel_match   HNZ|[BH]HZ                         # channels to check wfdisc start/end times and open records in stage table

# Only 4 adoption types supported: E, T, R, X.
# Do not change or add to list of codes below.
# atype_phrase can be modified

adoption_types &Arr{
# code  atype_phrase
E       EARN
T       OPERATOR TRANSITION
R       REFNET / BACKBONE
X       CEUSN/1N4
}

accepted_comm_types &Tbl{
DSL
WiFi
cable
cell modem
no comms
orb
orb2orb
regular internet
slink2orb
vsat
}

accepted_comm_providers &Tbl{
ATT
Cal_Ore tel
DSL Frontier
DoE
etc.
}


This parameter file maintains the default locations for various orbs and dbs such that they do not have to be repeated on the command line unless overrides are required.

EXAMPLE

There are four different modes of operation for this program: installation, update, removal, or adoption.

For a new installation: mk_dbops -I [-k] [-v] [-n] [-V vnet] [-m source_match] [-p pf] [-C cmdORB] [-P prelimORB] [-S statusORB] [-w prelimwfDB] [-s siteDB] dbopsdb snet sta certify_time comm_provider comm_type For a comms update: mk_dbops -U [-k] [-v] [-p pf] dbopsdb snet sta time_of_comm_change comm_provider [comm_type] ; For a station removal: mk_dbops -R [-k] [-v] [-p pf] [-W wfDB] [-s siteDB] dbopsdb snet sta decert_time For a station transition to regional network: mk_dbops -A [-k] [-v] [-p pf] [-W wfDB] [-s siteDB] dbopsdb snet sta decert_time

Installation

Update the comm, deployment, and dlsite table after installation of TA TEST a vsat site with comms provided by Wild Blue. Certification time is 12/10/2008 18:30 UTC.

% mk_dbops -I dbops/usarray TA TEST \
        "12/10/2008 18:30:00" "Wild Blue" vsat 

Update the comm, deployment, and dlsite table after installation of BB TST2 a cell modem site with comms provided by Alltel and part of the _US-TEST vitural network. Certification time is 12/10/2008 18:30 UTC. Use the prelim waveform database for the first data available time, deployment.time.

% mk_dbops -I dbops/usarray -V _US-TEST -w prelim/usarray \
	BB TST2 "12/10/2008 18:30:00" "Alltel" "cell modem" 

Update

Update the comm and dlsite tables after the communications provider for TA EXST changed from Verizon to ATT. Time new comms were established is 12/10/2008 18:30:00. Keep backups of the comm and dlsite tables.

% mk_dbops -k -U dbops/usarray TA EXST \
        "12/10/2008 18:30:00" ATT 

Removal

Update the comm, deployment and dlsite tables for TA DONE. Decertification (time metadata was passed along closing the station) was 12/10/2008 18:30:00. Keep backups of the comm, deployment and site tables.

% mk_dbops -k -R dbops/usarray TA DONE 2008:353:18:30:00 

Adoption/Transition

Update the adoption, comm, deployment and dlsite tables, after station TA ADPT is adopted by PNSN. Time station transitioned to non-standard TA opertations was 12/10/2008 18:30:00. Station was adopted by UW and called SNOW.

% mk_dbops -A dbops/usarray TA ADPT "12/10/2008 18:30:00" 

mk_dbops: Adoption type(E|T|R|X|-): T
mk_dbops: New station name(ADPT): SNOW
mk_dbops: New network code(TA): UW
mk_dbops: Newly responsible data center(????????): PNSN

Update the comm, deployment and dlsite tables, after transition of station TA ERNN. Time station transitioned to non-standard TA operations was 12/10/2008 18:30:00. Station is part of the EARN program and will continue to flow through the ANF with the same name.

% mk_dbops -A dbops/usarray TA ERNN "12/10/2008 18:30:00" 

mk_dbops: Adoption type(E|T|R|-): E
mk_dbops: New vnet code for EARN station(_XX-XXXXX): _US-EARN
mk_dbops: New station name(EARN):  
mk_dbops: New network code(TA):  
mk_dbops: Newly responsible data center(-): ANF

Update the comm and deployment tables of the usarray database and the comm and deployment table of the ceusn database, after transition of station TA A01A. Time station transitioned to N4/CEUSN operations was 03/13/2015 18:00:00.

% mk_dbops -n -A db/usarray TA A01A "03/13/2015 18:00:00" 

mk_dbops: Adoption type (E|T|R|X|-):  X
mk_dbops: New station name (A01A):
mk_dbops: New network code (TA):  N4
mk_dbops: Newly responsible data center (????????):  ANF
mk_dbops: New vnet code for  CEUSN/1N4  station (_XX-XXXXXXXX):  _CEUSN
mk_dbops: New database for deployment and comms (new/db):  n4/dbops/ceusn

DIAGNOSTICS

The parameter file has a channel sifter in it called chan_match. For instance, if you have BHZ channels in your metadata and the parameter file is set to use HHZ channels, you will see this error.

If you are working with a removal, -R, or an adoption, -A with an adoption type of T|transition or X|CEUSN/N4 conversion you need to have updated the dbmaster tables to indicate the station is closed.

If you are working on an installation, -I, the metadata for that station must be in the input database before running mk_dbops.

SEE ALSO

adoption(5)
comm(5)
deployment(5)
convert_comm(1)
convert_dlsite(1)
convert_deployment(1)
deployment2vnd(1)
mk_dmc_files(1)
q330_location(1)

BUGS AND CAVEATS

This script is firmly founded upon assumptions made in the operational model of your network. Testing has not been attempted for setups outside what is in use at the ANF. For instance, you may run across errors if you do not operate separate prelim and cmd orbs. Per agreed upon conventions: EARN stations retain their same snet_sta, TRANSITION stations may change their snet_sta, and REFNET/Backbone stations should retain their same snet_sta. If these conventions change, reprogramming, or at least further testing of the current script is necessary. If using -I mode with the default behavior to pull the oldest data time from the prelim orb, you must have permission to access the orb. If using -I or -U mode that checks the status orb, you must have permission to access the orb. User must have write access to dbops tables (adoption, comm, deployment, and dlsite). If a record cannot be added to a table the program dies. This could be a problem, for instance, if one table had a successful update (i.e. comm) and then the update to the deployment table fails. Prior to any updating, the script creates backups of all tables to be updated. They are called dbopsdb.table+ and are removed upon successful completion of the script unless -k is specified. When updating the dlsite table using -I mode, the q330_location code is run. As of the 4.11 release, this q330_location script subsets based on a single station. Earlier releases did not subset based on a single station, so all stations were reviewed and reported upon. This caused some problems with null values for commtype and provider if there were multiple stations that had not been added to the deployment table. Also note that q330_location has copious output. If dbmaster tables are not updated for a newly installed station, you will not be able to update the tables in -I mode. When running in -I mode, if the newly certified stations have no data in the default status_orb you will have to run q330_location(1) by hand. Alternatively, use the -S option to point at the orb that has the necessary .*/st packets, likely the prelimORB. When running in -U mode, if you only have duty cycle status or power type to update, you will still need to put in the comm provider in order to convince the script to update the comm table. The deployment table subset has been modified to include the vnet. This should deal with instances for example where you had a station operational as TA, it was adopted/transitioned to operations as N4, and then you have to close the station.

AUTHOR

Jennifer Eakins
ANF-IGPP-SIO-UCSD
jeakins@ucsd.edu
Antelope User Group Contributed Software
Printer icon