Commit 24e95345 authored by Trevor Cappallo's avatar Trevor Cappallo
Browse files

Merge in README

parents da0030e7 2230c565
Logging for Operations.
## Logging for Operations (or anything else)
Originally forked from BTS/Lightning on 05-Aug-2015.
(Originally forked from BTS/Lightning on 05-Aug-2015.)
---
### Quick start
1. git clone this, then run `python2.7 setup.py install --user`
2. in your top script, add the lines:
1. `import OperationalLogging as OpLog`
2. `OpLog.setup()` --- *this must come before your first call to `logging.basicConfig`*
That's all.
-----
### Detailed usage
(Not all of this has been tested yet.)
So long as you call `OperationalLogging.setup()` first, all subsequent `logging.basicConfig()`s in your code will be ignored. You only need to include `setup()` the once in your top-level script. Any logging calls deeper in your scripts should be handled by OpLog unless they explicitly provide a different logger name.
#### **Optional arguments to `setup()`**
* `config_file`: Filename of a .json file. It should contain a dict that will be read in as logging config using the keywords below. Any other arguments provided to `setup()` will override any options loaded from file.
* `logger_name`: Equivalent to using `getLogger(logger_name)`; the default of root/None is probably preferable.
* `log_filename`, `log_root_dir`: Log will be written to file at DEBUG level, saved as `LOG_ROOT_DIR/log/YYYY-MM-DD/log_filename.log`. `log_root_dir` defaults to current directory, and `log_filename` to scriptName.log.
* `from_address`: If email notifications are sent, this appears as the From field.
* `notify`: Dict containing lists of email addresses to notify at a given level, e.g. `{'ERROR': ['alice@example.com', 'bob@example.com'], 'CRITICAL': ['msze@aer.com']]}`. At first, email notification will only be (potentially) triggered on script exit.
* `log_level`: Overall log level lower bound. Strongly recommended to leave this at its default value of `logging.DEBUG`, so you have it all on disk.
* `console_level`: Log level for stdout if applicable. Defaults to `logging.INFO`.
* `log_format_file`, `date_format_file`: Equivalent to `fmt` and `datefmt` options as supplied to logging handler formatter, applies to the log file only.
* `log_format_console`, `date_format_console`: As above, but for stdout.
#### **log_shell()**
For logging `subprocess` calls, use: `OperationalLogging.log_shell(arg_list, **kwargs)`
In general, `OpLog.log_shell([args])` will behave like `subprocess.call([args])`. It routes stderr to stdout, and it will return the process's exit code. This actually calls `os.Popen` instead of `subprocess` which was necessary to get an unbuffered pipe to see console output in real time. Any extra arguments will be passed to `Popen`.
(This seems to be solid---even the multicore aggregation works.)
---
### Example output
Very much subject to change. The file output contains a little more info by default, e.g.
```
[2015-08-14 04:37:50][ INFO][p2565][OperationalLogging.py:190] Logging output from: PrepareUtils/scripts/prepare_HURSIM_Wind_driver.py --force aal18 2012102512 -v --dest /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512 /Production/Prepare/test_cases/Location/Test.upx --env /network/apps/prepare/v0.3/ --windows_host lex-vcwindev01 --windows_user msze
[2015-08-14 04:37:50][ INFO][p2565][OperationalLogging.py:191] Log writing to: /network/apps/prepare/v0.3/log/2015-08-14/prepare_HURSIM_Wind_driver.log
[2015-08-14 04:37:50][ INFO][p2565][OperationalLogging.py:192] Process id 2565, parent process id 2562
[2015-08-14 04:37:50][ INFO][p2565][OperationalLogging.py:193] ------------------------------------------------------------
[2015-08-14 04:37:50][ INFO][p2565][prepare_HURSIM_Wind_driver.py:71] environ: /network/apps/prepare/v0.3/
[2015-08-14 04:37:50][ INFO][p2565][Hursim.py:33] {'host': 'lex-vcwindev01', 'password': None, 'user': 'prepare'}
[2015-08-14 04:37:50][ INFO][p2565][Simulation.py:128] generate simulated ensemble
[2015-08-14 04:37:50][DEBUG][p2565][EnsembleGeneration.py:76] Ensemble Generation Runtime
```
as compared to the following, which is the console output:
```
[prepare@lex-prepare01|04:37] /network/apps/prepare/v0.3> ./run_hursim.sh 18 2012102512
/usr/local/CentOS6/lib/python2.7/site-packages/pycrypto-2.6.1-py2.7-linux-x86_64.egg/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
04:37:50 ( INFO) [OperationalLogging.py:190] Logging output from: PrepareUtils/scripts/prepare_HURSIM_Wind_driver.py --force aal18 2012102512 -v --dest /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512 /Production/Prepare/test_cases/Location/Test.upx --env /ne
twork/apps/prepare/v0.3/ --windows_host lex-vcwindev01 --windows_user msze
04:37:50 ( INFO) [OperationalLogging.py:191] Log writing to: /network/apps/prepare/v0.3/log/2015-08-14/prepare_HURSIM_Wind_driver.log
04:37:50 ( INFO) [OperationalLogging.py:192] Process id 2565, parent process id 2562
04:37:50 ( INFO) [OperationalLogging.py:193] ------------------------------------------------------------
04:37:50 ( INFO) [prepare_HURSIM_Wind_driver.py:71] environ: /network/apps/prepare/v0.3/
04:37:50 ( INFO) [Hursim.py:33] {'host': 'lex-vcwindev01', 'password': None, 'user': 'prepare'}
04:37:50 ( INFO) [Simulation.py:128] generate simulated ensemble
04:37:50 (DEBUG) [EnsembleGeneration.py:76] Ensemble Generation Runtime
&userin
inputArchive = './atcf/AAL' ! archive of operational ensemble data files (formatted by storm #)
inputFile = './data/aalSSYYYY_MMDDHH.dat' ! input file containing operational ensemble tracks (the ./data part is hardcoded)
bIncludeOrig = .TRUE. ! set to TRUE to include original operational ensemble in simulated output
nSampleSize = 500 ! number of tracks to be generated from ensemble (draws)
fPertIntn = 1.0 ! magnitude of max perturbation of intensity along track (uniform in STDEV)
fPertSkew = 0.0 ! degree of skewness in perturbation of intensity along track (0=no skewness)
fRadInflu = 500.0 ! radius of influence for intensity estimation points
fDistThres = 100.0 ! distance from land to which storm would produce no loss assuming max winds of 40mph
fWindThres = 35.0 ! wind below which storm is considered not possible to produce loss
outputEns = 'operationEnsemble' ! output file rootname (extraction of the operational ensemble)
outputSim = 'simulatedEnsemble' ! output file rootname (simulated and perturbed from operational ensemble)
outputAudit = 'auditFile' ! output file rootname for runtime log (including random draws)
outputTransPath = '../Data/Ensemble/YYYY/' ! output path for all output to be transfered upon completion ('0' for no transfer)
bPurgeData = .TRUE. ! set to TRUE to purge operational model extraction used in this analysis
bEstimateRmax = .TRUE. ! set to TRUE to use model HU and TS radius to estimate Rmax (otherwise set to 0)
&end
04:37:50 (DEBUG) [EnsembleGeneration.py:115] executing EnsembleGeneration ['/network/apps/prepare/v0.3/EnsembleGeneration/ensembleTracks', '/Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512/EnsembleGeneration_HURSIM/runtime_cfg', '18', '2012', '10', '25', '12'
]
04:37:50 ( INFO) [OperationalLogging.py:235] Starting child process of ppid=2562
04:37:50 ( INFO) [OperationalLogging.py:236] Executing: /network/apps/prepare/v0.3/EnsembleGeneration/ensembleTracks /Production/Prepare/prepare_validation_testing/log_test/aal18_2012102512/EnsembleGeneration_HURSIM/runtime_cfg 18 2012 10 25 12
04:37:50 |-
04:37:50 |- ============ START OF ENSEMBLE PROCESSING
04:37:50 |-
04:37:50 |- ...successfully read land mask point file
04:37:50 |-
04:37:50 |- Note: Historical data is not available for this storm. No comparison will be made to observed track.
04:37:50 |-
04:37:50 |- INPUT FILE USED FOR THIS ANALYSIS: ./data/aal182012_102512.dat
04:37:50 |-
04:37:51 |- Opening inputFile: ./data/aal182012_102512.dat
04:37:51 |-
04:37:51 |- STARTING POSITION BASED ON OPERATIONAL ENSEMBLE INITIAL CONDITIONS: ( 21.66, -75.55)
04:37:51 |-
04:37:51 |- Number of unique model tracks found: 100
04:37:51 |- Number of times read (processed): 1862 ( 43)
04:37:51 |- [DEBUG] Note: model (CARQ) removed from the simulation likely due to incomplete input data
04:37:51 |- [DEBUG] Note: model (BAMD) removed from the simulation likely due to incomplete input data
```
...etc.
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment