Logging

conversion cockpit's doc/logs/*.txt

A log file is created every time the conversion trigger is pulled. It is placed in the conversion cockpit's doc/log/ directory.

$CSV2RDF4LOD_HOME/bin/convert.sh and $CSV2RDF4LOD_HOME/bin/convert-aggregate.sh always log messages to:

CSV2RDF4LOD_LOG="doc/logs/csv2rdf4lod_log_e${eID}_`date +%Y-%m-%dT%H_%M_%S`.txt"

The number of logs in this directory is asserted as conversion:num_invocation_logs in the aggregated data dump Turtle file publish/<dataset-id>-<version-id>.ttl

$ cr-pwd.sh 
source/

$ cr-trim-logs.sh
...
...
========== source/data-gov/1554/version/2011-Jan-12 ========================================

319M doc/logs total
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_37_26.txt   24 -> 24
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_39_16.txt   24 -> 24
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_42_29.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_42_44.txt   328 -> 328
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_44_41.txt   24 -> 24
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_44_48.txt   328 -> 328
doc/logs/csv2rdf4lod_log_e1_2011-03-29T13_59_12.txt   19344 -> 19344
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_00_30.txt   19344 -> 19344
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_12_31.txt   19344 -> 19344
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_14_59.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_17_32.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_19_19.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_21_26.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_24_47.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_25_10.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_34_23.txt   28 -> 28
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_35_41.txt   19904 -> 19904
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_41_59.txt   19908 -> 19908
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_44_37.txt   4
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_51_12.txt   19908 -> 19908
doc/logs/csv2rdf4lod_log_e1_2011-03-29T14_52_55.txt   19916 -> 19916
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_08_18.txt   4304 -> 4304
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_08_38.txt   19916 -> 19916
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_11_07.txt   40 -> 40
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_12_19.txt   40 -> 40
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_14_22.txt   40 -> 40
doc/logs/csv2rdf4lod_log_e1_2011-03-29T15_56_35.txt   44 -> 44
doc/logs/csv2rdf4lod_log_e1_2011-04-12T08_47_21.txt   44 -> 44
doc/logs/csv2rdf4lod_log_raw_2011-03-21T14_20_41.txt   40 -> 40

Note: did not trim logs. Use cr-trim-logs.sh -w to modify doc/logs/*.txt
...
...

$ cr-trim-logs.sh | grep total
604K doc/logs total
116K doc/logs total
319M doc/logs total
216K doc/logs total
136K doc/logs total
16K doc/logs total
12K doc/logs total
24K doc/logs total
24K doc/logs total

Loading and Deleting a graph.

These are in tmp b/c Virtuoso needs permission to write.

$CSV2RDF4LOD_HOME/bin/util/virtuoso/vload stores logs to $CSV2RDF4LOD_HOME/tmp/vload-tmp/input-files/*.log with the latest at $CSV2RDF4LOD_HOME/tmp/vload-tmp/input-files/latest.log.

$CSV2RDF4LOD_HOME/bin/util/virtuoso/vdelete stores logs to $CSV2RDF4LOD_HOME/tmp/vdelete-tmp/*.log with the latest at $CSV2RDF4LOD_HOME/tmp/vdelete-tmp/latest.log.

populate-endpoint.sh

populate-endpoint.sh needs to be generalized beyond LOGD. It loads metadata from all conversions into a named graph and caches query results to static files to reduce endpoint load when supporting a web site.

${CSV2RDF4LOD_HOME}/log/populate-endpoint.sh/*.log

Turning on the converter's logging

In debugging situations, I might have you turn this on. It should rarely be needed.

The Java implementation uses java.util.logging to log.

Turning logging on is parameterized by the CSV2RDF4LOD_CONVERT_DEBUG_LEVEL environment variable and takes affect within $CSV2RDF4LOD_HOME/bin/convert.sh:

javaprops="-Djava.util.logging.config.file=$CSV2RDF4LOD_HOME/bin/logging/finest.properties"
#javaprops=""

So,

$ export CSV2RDF4LOD_CONVERT_DEBUG_LEVEL=finer

other valid values include fine, finer, and finest.

CSV2RDF4LOD_HOME/bin/logging/ contains fine, finer, and finest.properties.

(If you REALLY wanna get your hands dirty, add your.properties in CSV2RDF4LOD_HOME/bin/logging/ and set your DEBUG_LEVEL to your.)