Skip to content

Conversion process phase: csv ify

Jump to bottom
timrdf edited this page Oct 1, 2011 · 41 revisions

[up](Conversion process phases)

(this should be renamed to "Conversion process phase: tweak source data")

What's first?

The conversion cockpit's manual/ directory should be used to store the results of any manual tweaks of original source/ data. If no tweaks need to be made, the data can be converted directly from source/.

source/ vs. manual/

As discussed in Conversion process phase: retrieve, a conversion cockpit's source/ directory holds an unmodified copy of the data that you received from the source (e.g. rpi-edu-lebot, whitehouse-gov). If you used pcurl.sh, then you also captured the provenance justifying the file on disk by citing the authoritative URL from which it came. You SHOULD NOT modify any of the files you get from your source. If you need to, make a modified copy in manual/.

What if it already is a tabular file?

If the file that you obtained is already in the CSV format, then you do NOT need to duplicate the file into manual/. You can create the conversion trigger directly from the files in source/. Check out the next phase Conversion process phase: create conversion trigger for more about that.

Associating tweaked manual/ files back to their originals in source/

If we manually tweak any of the original data files that we retrieved, we are adding another step in the chain from final results back to their original. Associating the files we create to their originals ensures that we can provide a level of transparency.

A simple example of this is to convert a tab or pipe delimited file into csv:

bash-3.2$ cat source/some.tsv | sed -e 's/^/"/' -e 's/|/","/g' -e 's/$/"/' > manual/some.tsv.csv

NOTE changing tabular cell delimiters can be done with enhancement parameters and should NOT be done manually.

Here, we are making a new file in manual/ that parallels the original file in source/. When naming the new file, we use a convention of appending the new file extension (.csv) to the entire file name of the original file (some.tsv). This helps others trace the lineage using only file names and without any overhead of digging through metadata. To be explicit, so also generate the metadata by running:

bash-3.2$ justify.sh source/some.tsv manual/some.tsv.csv redelimit

This creates manual/some.tsv.csv.pml.ttl and records that manual/some.tsv.csv came from source/some.tsv using a method known as redelimit (and that you did it).

What's next?

Clone this wiki locally