Installation¶
S3QL depends on several other programs and libraries that have to be installed first. The best method to satisfy these dependencies depends on your distribution.
Dependencies¶
The following is a list of the programs and libraries required for running S3QL. Generally, you should first check if your distribution already provides a suitable packages and only install from source if that is not the case.
Kernel: Linux 3.9 or newer.
The psmisc utilities.
SQLite version 3.7.0 or newer. SQLite has to be installed as a shared library with development headers.
Python 3.7 or newer. Make sure to also install the development headers.
The following Python modules:
setuptools, version 1.0 or newer.
apsw, version 3.42.0 or newer.
trio, version 0.15 or newer.
pyfuse3, any version between 3.2.0 (inclusive) and 4.0 (exclusive)
pytest, version 3.7 or newer (optional, to run unit tests)
systemd (optional, for enabling systemd support). Do not install the module from PyPi, this is from a third-party developer and incompatible with the official module from the systemd developers.
requests (optional, required for OAuth2 authentication with Google Storage)
google-auth (optional, required for ADC authentication with Google Storage)
google-auth-oauthlib (optional, required for browser-based authentication with Google Storage)
pytest_trio (optional, to run unit tests)
To check if a specific module
<module>
is installed, executepython3 -c 'import <module>; print(<module>.__version__)'
. This will result in anModuleNotFoundError
if the module is not installed, and will print the installed version if the module is installed.
Installing S3QL¶
To build and install S3QL, first download the release tarball from GitHub. Then validate the tarball using signify:
signify -V -m s3ql-XX.tar.gz -p s3ql-XX.pub
The s3ql-XX.pub
file needs to be obtained from a trustworthy source (it contains the
signing key). Each S3QL release contains the signing key for the release after it in the
signify
directory, so you only need to manually acquire this file once when you install
S3QL for the first time).
After validating the tarball, unpack it and change into the newly created s3ql-X.Y.Z
directory. Then run:
python3 setup.py build_ext --inplace
to build S3QL and
python3 -m pytest tests/
to run a self-test. If this fails, ask for help on the mailing list or report a bug in the issue tracker.
Now you have three options:
You can run the S3QL commands from the
bin/
directory.You can install S3QL system-wide for all users. To do that, you have to run
sudo python3 setup.py install
.You can install S3QL into
~/.local
by executingpython3 setup.py install --user
. In this case you should make sure that~/.local/bin
is in your$PATH
environment variable.
Development Version¶
If you have checked out the unstable development version from the Git repository, a bit more effort is required. You’ll also need:
Version 0.28.1 or newer of the Cython compiler.
Version 1.2b1 or newer of the Sphinx document processor.
With these additional dependencies installed, S3QL can be build and tested with
python3 setup.py build_cython
python3 setup.py build_ext --inplace
python3 -m pytest tests
Note that when building from the Git repository, building and testing is done with several
additional checks. This may cause compilation and/or tests to fail even though there are
no problems with functionality. For example, any use of functions that are scheduled for
deprecation in future Python version will cause tests to fail. If you would rather just
check for functionality, you can delete the MANIFEST.in
file. In that case, the
build system will behave as it does for a regular release.
The HTML and PDF documentation can be generated with
./build_docs.sh
(cd doc/pdf && make)
and S3QL can be installed as usual with
python3 setup.py install [--user]
Running tests requiring remote servers¶
By default, tests requiring a connection to a remote storage backend
are skipped. If you would like to run these tests too (which is always
a good idea), you have to create additional entries in your
~/.s3ql/authinfo2
file that tell S3QL what server and credentials to
use for these tests. These entries have the following form:
[<BACKEND>-test]
backend-login: <user>
backend-password: <password>
test-fs: <storage-url>
Here <BACKEND> specifies the backend that you want to test (e.g. s3, s3c, gs, or swift), <user> and <password> are the backend authentication credentials, and <storage-url> specifies the full storage URL that will be used for testing. Any existing S3QL file system in this storage URL will be destroyed during testing.
For example, to run tests that need connection to a Google Storage server, you would add something like
[gs-test]
backend-login: GOOGIGWLONT238MD7HZ4
backend-password: rmEbstjscoeunt1249oes1298gauidbs3hl
test-fs: gs://joes-gs-bucket/s3ql_tests/