Commit 2230c565 authored by Trevor Cappallo's avatar Trevor Cappallo
Browse files

Extensive README

parent d6103181
Logging for Operations.
## Logging for Operations (or anything else)
Originally forked from BTS/Lightning on 05-Aug-2015.
(Originally forked from BTS/Lightning on 05-Aug-2015.)
### Quick start
1. git clone this, then run `python2.7 install --user`
2. in your top script, add the lines:
1. `import OperationalLogging as OpLog`
2. `OpLog.setup()` --- *this must come before your first call to `logging.basicConfig`*
That's all.
### Detailed usage
(Not all of this has been tested yet.)
So long as you call `OperationalLogging.setup()` first, all subsequent `logging.basicConfig()`s in your code will be ignored. You only need to include `setup()` the once in your top-level script. Any logging calls deeper in your scripts should be handled by OpLog unless they explicitly provide a different logger name.
#### **Optional arguments to `setup()`**
* `config_file`: Filename of a .json file. It should contain a dict that will be read in as logging config using the keywords below. Any other arguments provided to `setup()` will override any options loaded from file.
* `logger_name`: Equivalent to using `getLogger(logger_name)`; the default of root/None is probably preferable.
* `log_filename`, `log_root_dir`: Log will be written to file at DEBUG level, saved as `LOG_ROOT_DIR/log/YYYY-MM-DD/log_filename.log`. `log_root_dir` defaults to current directory, and `log_filename` to scriptName.log.
* `from_address`: If email notifications are sent, this appears as the From field.
* `notify`: Dict containing lists of email addresses to notify at a given level, e.g. `{'ERROR': ['', ''], 'CRITICAL': ['']]}`. At first, email notification will only be (potentially) triggered on script exit.
* `log_level`: Overall log level lower bound. Strongly recommended to leave this at its default value of `logging.DEBUG`, so you have it all on disk.
* `console_level`: Log level for stdout if applicable. Defaults to `logging.INFO`.
* `log_format_file`, `date_format_file`: Equivalent to `fmt` and `datefmt` options as supplied to logging handler formatter, applies to the log file only.
* `log_format_console`, `date_format_console`: As above, but for stdout.
#### **log_shell()**
For logging `subprocess` calls, use: `OperationalLogging.log_shell(arg_list, **kwargs)`
In general, `OpLog.log_shell([args])` will behave like `[args])`. It routes stderr to stdout, and it will return the process's exit code. This actually calls `os.Popen` instead of `subprocess` which was necessary to get an unbuffered pipe to see console output in real time. Any extra arguments will be passed to `Popen`.
(This seems to be solid---even the multicore aggregation works.)
### Example output
Very much subject to change. The file output contains a little more info by default, e.g.
[2015-08-14 04:37:50][ INFO][p2565][] Logging output from: PrepareUtils/scripts/ --force aal18 2012102512 -v --dest /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512 /Production/Prepare/test_cases/Location/Test.upx --env /network/apps/prepare/v0.3/ --windows_host lex-vcwindev01 --windows_user msze
[2015-08-14 04:37:50][ INFO][p2565][] Log writing to: /network/apps/prepare/v0.3/log/2015-08-14/prepare_HURSIM_Wind_driver.log
[2015-08-14 04:37:50][ INFO][p2565][] Process id 2565, parent process id 2562
[2015-08-14 04:37:50][ INFO][p2565][] ------------------------------------------------------------
[2015-08-14 04:37:50][ INFO][p2565][] environ: /network/apps/prepare/v0.3/
[2015-08-14 04:37:50][ INFO][p2565][] {'host': 'lex-vcwindev01', 'password': None, 'user': 'prepare'}
[2015-08-14 04:37:50][ INFO][p2565][] generate simulated ensemble
[2015-08-14 04:37:50][DEBUG][p2565][] Ensemble Generation Runtime
as compared to the following, which is the console output:
[prepare@lex-prepare01|04:37] /network/apps/prepare/v0.3> ./ 18 2012102512
/usr/local/CentOS6/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg/Crypto/Util/ PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
04:37:50 ( INFO) [] Logging output from: PrepareUtils/scripts/ --force aal18 2012102512 -v --dest /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512 /Production/Prepare/test_cases/Location/Test.upx --env /ne
twork/apps/prepare/v0.3/ --windows_host lex-vcwindev01 --windows_user msze
04:37:50 ( INFO) [] Log writing to: /network/apps/prepare/v0.3/log/2015-08-14/prepare_HURSIM_Wind_driver.log
04:37:50 ( INFO) [] Process id 2565, parent process id 2562
04:37:50 ( INFO) [] ------------------------------------------------------------
04:37:50 ( INFO) [] environ: /network/apps/prepare/v0.3/
04:37:50 ( INFO) [] {'host': 'lex-vcwindev01', 'password': None, 'user': 'prepare'}
04:37:50 ( INFO) [] generate simulated ensemble
04:37:50 (DEBUG) [] Ensemble Generation Runtime
inputArchive = './atcf/AAL' ! archive of operational ensemble data files (formatted by storm #)
inputFile = './data/aalSSYYYY_MMDDHH.dat' ! input file containing operational ensemble tracks (the ./data part is hardcoded)
bIncludeOrig = .TRUE. ! set to TRUE to include original operational ensemble in simulated output
nSampleSize = 500 ! number of tracks to be generated from ensemble (draws)
fPertIntn = 1.0 ! magnitude of max perturbation of intensity along track (uniform in STDEV)
fPertSkew = 0.0 ! degree of skewness in perturbation of intensity along track (0=no skewness)
fRadInflu = 500.0 ! radius of influence for intensity estimation points
fDistThres = 100.0 ! distance from land to which storm would produce no loss assuming max winds of 40mph
fWindThres = 35.0 ! wind below which storm is considered not possible to produce loss
outputEns = 'operationEnsemble' ! output file rootname (extraction of the operational ensemble)
outputSim = 'simulatedEnsemble' ! output file rootname (simulated and perturbed from operational ensemble)
outputAudit = 'auditFile' ! output file rootname for runtime log (including random draws)
outputTransPath = '../Data/Ensemble/YYYY/' ! output path for all output to be transfered upon completion ('0' for no transfer)
bPurgeData = .TRUE. ! set to TRUE to purge operational model extraction used in this analysis
bEstimateRmax = .TRUE. ! set to TRUE to use model HU and TS radius to estimate Rmax (otherwise set to 0)
04:37:50 (DEBUG) [] executing EnsembleGeneration ['/network/apps/prepare/v0.3/EnsembleGeneration/ensembleTracks', '/Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512/EnsembleGeneration_HURSIM/runtime_cfg', '18', '2012', '10', '25', '12'
04:37:50 ( INFO) [] Starting child process of ppid=2562
04:37:50 ( INFO) [] Executing: /network/apps/prepare/v0.3/EnsembleGeneration/ensembleTracks /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512/EnsembleGeneration_HURSIM/runtime_cfg 18 2012 10 25 12
04:37:50 |-
04:37:50 |- ============ START OF ENSEMBLE PROCESSING
04:37:50 |-
04:37:50 |- ...successfully read land mask point file
04:37:50 |-
04:37:50 |- Note: Historical data is not available for this storm. No comparison will be made to observed track.
04:37:50 |-
04:37:50 |- INPUT FILE USED FOR THIS ANALYSIS: ./data/aal182012_102512.dat
04:37:50 |-
04:37:51 |- Opening inputFile: ./data/aal182012_102512.dat
04:37:51 |-
04:37:51 |-
04:37:51 |- Number of unique model tracks found: 100
04:37:51 |- Number of times read (processed): 1862 ( 43)
04:37:51 |- [DEBUG] Note: model (CARQ) removed from the simulation likely due to incomplete input data
04:37:51 |- [DEBUG] Note: model (BAMD) removed from the simulation likely due to incomplete input data
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment