Merge branch 'develop-64' into develop

Author: LaurensRietveld

.gitignore

@@ -54,8 +54,11 @@ stamp-h1
 
 *.hdt
 *.hdt.index
+*.nq
 *.nt
 *.rdf
+*.trig
+*.ttl
 *.a
 **/examples/*
 **/tests/*

README.md

@@ -1,77 +1,130 @@
 [![Join the chat at https://gitter.im/rdfhdt](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/rdfhdt)
 [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.580298.svg)](https://doi.org/10.5281/zenodo.580298)
 
-# C++ library for the HDT triple format
+# C++ implementation of the HDT compression format
 
-HDT keeps big RDF datasets compressed while maintaining efficient search and browse operations.
+Header Dictionary Triples (HDT) is a compression format for RDF data
+that can also be queried for Triple Patterns.
 
 ## Getting Started
+
 ### Prerequisites
 
-The implementation has the following dependencies:
-- [Serd v0.28+](http://drobilla.net/software/serd/) This enables importing RDF data in the Turtle and N-Triples serialization formats specifically. The dependency is activated by default.
-- [libz](http://www.zlib.net/) Enables loading N-Triples files compressed with GZIP (e.g., `file.nt.gz`) and gzipped HDTs (`file.hdt.gz`). The dependency is activated by default.
+In order to compile this library, you need to have the following
+dependencies installed:
+
+- [GNU Autoconf](https://www.gnu.org/software/autoconf/autoconf.html)
+
+  - `sudo apt install autoconf` on Debian-based distros (e.g., Ubuntu)
+  - `sudo dnf install autoconf` on Red Hat-based distros (e.g.,
+    Fedora)
+
+- [GNU Libtool](https://www.gnu.org/software/libtool/)
 
-The installation process has the following dependencies:
+  - `sudo apt install libtool` on Debian-based distros (e.g., Ubuntu)
+  - `sudo dnf install libtool` on Red Hat-based distros (e.g., Fedora)
+
+- [GNU zip (gzip)](http://www.zlib.net/) Allows GNU zipped RDF input
+  files to be ingested, and allows GNU zipped HDT files to be loaded.
+
+  - `sudo apt install gzip` on Debian-based distros (e.g., Ubuntu)
+  - `sudo dnf install gzip` on Red Hat-based distros (e.g., Fedora)
 
-- [autoconf](https://www.gnu.org/software/autoconf/autoconf.html)
-- [libtool](https://www.gnu.org/software/libtool/)
 - [pkg-config](https://www.freedesktop.org/wiki/Software/pkg-config/)
+  A helper tool for compiling applications and libraries.
+
+  - `sudo apt install pkg-config` on Debian-based distros (e.g.,
+    Ubuntu)
+  - `sudo dnf install pkgconf-pkg-config` on Red Hat-based distros
+    (e.g., Fedora)
 
-The following commands should install all packages:
+- [Serd v0.28+](https://github.com/drobilla/serd) The default parser
+  that is used to process RDF input files.  It supports the N-Quads,
+  N-Triples, TriG, and Turtle serialization formats.
 
-    sudo apt-get update
-    sudo apt-get install autoconf libtool pkg-config
+  - `sudo apt install libserd-0-0 libserd-dev` on Debian-based distros
+    (e.g., Ubuntu)
+  - `sudo dnf install serd serd-devel` on Red Hat-based distros (e.g.,
+    Fedora)
 
-### Installing
+### Installation
 
-To compile and install, run the following commands under the directory `hdt-cpp`. This will generate the library and tools.
+To compile and install, run the following commands under the directory
+`hdt-cpp`.  This will also compile and install some handy tools.
 
-First run the following script to generate all necessary installation files with autotools:
+```
+./autogen.sh
+./configure
+make -j2
+sudo make install
+```
 
-    ./autogen.sh
+### Installation issues
 
-Then, run:
+Sometimes, the above instructions do not result in a working HDT
+installation.  This section enumerates common issues and their
+workaround.
 
-    ./configure
-    make -j2
+#### `./configure` cannot find Serd
 
-If you get the error `No package 'serd-0' found` with `./configure`, you must install Serd manually (the last command may require `sudo`):
+While running `./configure` you get a message similar to the
+following:
 
-```shell
-wget https://github.com/drobilla/serd/archive/v0.28.0.tar.gz &&\
-tar -xvzf *.tar.gz && rm *.tar.gz && cd serd-* &&\
-./waf configure && ./waf &&\
-./waf install
 ```
-## Running
+Package 'serd-0', required by 'virtual:world', not found
+```
 
-After building, these are the typical operations that you will perform:
+This means that `./configure` cannot find the location of the
+`serd-0.pc` file on your computer.  You have to find this location
+yourself, e.g., in the following way:
 
-- Convert your RDF data to HDT:
+```sh
+find /usr/ -name serd-0.pc
+```
 
-    NB: the input stream is assumed to be valid RDF, so you should validate your data before feeding it into rdf2hdt.
+Once you have found the directory containing the `serd-0.pc` file, you
+have to inform the `./configure` script about this location by setting
+the following environment variable (where directory
+`/usr/local/lib/pkgconfig/` is adapted to your situation):
 
-    ```
-    $ libhdt/tools/rdf2hdt data/test.nt data/test.hdt
-    ```
+```sh
+export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/
+```
 
-- Create only the index of an HDT file:
+## Using HDT
 
-    ```
-    $ libhdt/tools/hdtSearch -q 0 data/test.hdt
-    ```
+After compiling and installing, you can use the handy tools that are
+located in `hdt-cpp/libhdt/tools`.  We show some common tasks that can
+be performed with these tools.
 
-- Convert an HDT to another RDF serialization format, such as N-Triples:
+### RDF-2-HDT: Creating an HDT
 
-    ```
-    $ libhdt/tools/hdt2rdf data/test.hdt data/test.hdtexport.nt
-    ```
+HDT files can only be created for standards-compliant RDF input files.
+If your input file is not standards-compliant RDF, it is not possible
+to create an HDT files out of it.
 
-- Open a terminal to search triple patterns within an HDT file:
+```
+$ ./rdf2hdt data.nt data.hdt
+```
 
-    ```
-    $ libhdt/tools/hdtSearch data/test.hdt
+### HDT-2-RDF: Exporting an HDT
+
+You can export an HDT file to an RDF file in one of the supported
+serialization formats (currently: N-Quads, N-Triples, TriG, and
+Turtle).  The default serialization format for exporting is N-Triples.
+
+```
+$ ./hdt2rdf data.hdt data.nt
+```
+
+### Querying for Triple Patterns
+
+You can issue Triple Pattern (TP) queries in the terminal by
+specifying a subject, predicate, and/or object term.  The questions
+mark (`?`) denotes an uninstantiated term.  For example, you can
+retrieve _all_ the triples by querying for the TP `? ? ?`:
+
+    $ ./hdtSearch data.hdt
     >> ? ? ?
     http://example.org/uri3 http://example.org/predicate3 http://example.org/uri4
     http://example.org/uri3 http://example.org/predicate3 http://example.org/uri5
@@ -91,24 +144,36 @@ After building, these are the typical operations that you will perform:
     2 results shown.
 
     >> exit
-    ```
 
-- Extract the Header of an HDT file:
+### Exporting the header
 
-    ```
-    $ libhdt/tools/hdtInfo data/test.hdt > header.nt
-    ```
+The header component of an HDT contains metadata describing the data
+contained in the HDT, as well as the creation metadata about the HDT
+itself.  The contents of the header can be exported to an N-Triples
+file:
 
-- Replace the Header of an HDT file with a new one. For example, by editing the existing one as extracted using `hdtInfo`:
+```
+$ ./hdtInfo data.hdt > header.nt
+```
+
+### Replacing the Header
 
-    ```
-    $ libhdt/tools/replaceHeader data/test.hdt data/testOutput.hdt newHeader.nt
-    ```
+It can be useful to update the header information of an HDT.  This can
+be done by generating a new HDT file (`new.hdt`) out of an existing
+HDT file (`old.hdt`) and an N-Triples file (`new-header.nt`) that
+contains the new header information:
+
+```
+$ ./replaceHeader old.hdt new.hdt new-header.nt
+```
 
 ## Contributing
 
-Contributions and PRs should be sent to the `develop` branch, and not to `master`.
+Contributions are welcome!  Please base your contributions and pull
+requests (PRs) on the `develop` branch, and not on the `master`
+branch.
 
 ## License
 
-`hdt-cpp` is free software licensed as GNU Lesser General Public License. See `libhdt/COPYRIGHT`
+`hdt-cpp` is free software licensed as GNU Lesser General Public
+License (GPL). See `libhdt/COPYRIGHT`.

configure.ac

@@ -28,7 +28,7 @@ AC_MSG_CHECKING(whether to enable optimizations)
 AC_ARG_ENABLE([optimization],
   AS_HELP_STRING([--enable-optimization],[Build library with optimization parameters [default=yes]]),
   [AC_MSG_RESULT(${enableval})],
-  [CXXFLAGS="${CXXFLAGS} -g -O2"]
+  [CXXFLAGS="${CXXFLAGS} -g -O2 -std=c++11"]
   [AC_MSG_RESULT(yes)])
 
 AC_MSG_CHECKING(whether to build libcds)

hdt-it/hdtcachedinfo.cpp

@@ -105,17 +105,17 @@ void HDTCachedInfo::generateMatrix(hdt::ProgressListener *listener)
 
 }
 
-Color * HDTCachedInfo::getPredicateColor(unsigned int npred)
+Color * HDTCachedInfo::getPredicateColor(size_t npred)
 {
     return &predicateColors[npred];
 }
 
-unsigned int HDTCachedInfo::getPredicateUsages(unsigned int predicate)
+size_t HDTCachedInfo::getPredicateUsages(size_t predicate)
 {
     return predicateCount[predicate];
 }
 
-unsigned int HDTCachedInfo::getMaxPredicateCount()
+size_t HDTCachedInfo::getMaxPredicateCount()
 {
     return maxPredicateCount;
 }
@@ -130,8 +130,8 @@ void HDTCachedInfo::save(QString &fileName, hdt::ProgressListener *listener)
 	// Only save info of files bigger than 2M triples. Otherwise is fast to create from scratch.
 	if(hdt->getTriples()->getNumberOfElements()>2000000) {
 	    std::ofstream out(fileName.toLatin1(), ios::binary);
-	    unsigned int numTriples = triples.size();
-	    out.write((char *)&numTriples, sizeof(unsigned int));
+        uint64_t numTriples = triples.size();
+        out.write((char *)&numTriples, sizeof(uint64_t));
 	    out.write((char *)&triples[0], sizeof(hdt::TripleID)*numTriples);
 	    out.close();
 	}
@@ -143,8 +143,8 @@ void HDTCachedInfo::load(QString &fileName, hdt::ProgressListener *listener)
 
     std::ifstream in(fileName.toLatin1(), ios::binary);
     if(in.good()) {
-        unsigned int numTriples;
-        in.read((char *)&numTriples, sizeof(unsigned int));
+        uint64_t numTriples;
+        in.read((char *)&numTriples, sizeof(uint64_t));
         triples.resize(numTriples);
         in.read((char *)&triples[0], sizeof(hdt::TripleID)*numTriples);
         in.close();

hdt-it/hdtcachedinfo.hpp

@@ -17,15 +17,15 @@ class HDTCachedInfo
     vector<hdt::TripleID> triples;
 
     vector<Color> predicateColors;
-    unsigned int maxPredicateCount;
-    vector<unsigned int> predicateCount;
+    size_t maxPredicateCount;
+    vector<size_t> predicateCount;
 
 public:
     HDTCachedInfo(hdt::HDT *hdt);
 
-    Color *getPredicateColor(unsigned int npred);
-    unsigned int getPredicateUsages(unsigned int predicate);
-    unsigned int getMaxPredicateCount();
+    Color *getPredicateColor(size_t npred);
+    size_t getPredicateUsages(size_t predicate);
+    size_t getMaxPredicateCount();
     vector<hdt::TripleID> &getTriples();
 
     void generateGeneralInfo(hdt::ProgressListener *listener=NULL);

hdt-it/headermodel.cpp

@@ -17,7 +17,7 @@ HeaderModel::~HeaderModel()
 int HeaderModel::rowCount(const QModelIndex &parent) const
 {
     if(hdtController->hasHDT()) {
-        return hdtController->getHDT()->getHeader()->getNumberOfElements();
+        return (int)hdtController->getHDT()->getHeader()->getNumberOfElements();
     }
     return 0;
 }