Datalad x UHH Object Storage

How to use the University of Hamburg’s Object Storage as a DataLad special remote

Object Storage website

About

• ❓ What: An Object Storage. Like Amazon S3 but by Cloudian.
• 💡 Purpose: For example, configure the object storage as a DataLad special remote
• 🔑 Access: Send an email to the RRZ-ServiceLine
• DataLad Handbook: “Walk-through: Amazon S3 as a special remote”
• 📚 LZS documentation documents
  • “LZS - First Steps” (in German 🇩🇪)
  • “LZS S3 Identity And Access Management” (in German 🇩🇪)
  • “Cloudian AWS CLI Guide” (in English 🇺🇸)

UHH Object Storage as a DataLad special remote

This section describes how to use the UHH Object Storage as a DataLad special remote.

Instructions may be incomplete!

The instructions below so far only document deviations from the DataLad Handbook walk-through “Amazon S3 as a special remote”

1. Follow the walk-through “Amazon S3 as a special remote” in the DataLad Handbook
2. Use the following command when initializing the S3 special remote:

git annex initremote public-s3 type=S3 encryption=none \
bucket=$BUCKET public=yes datacenter=EU \
host=s3-uhh.lzs.uni-hamburg.de protocol=https port=443 autoenable=true

3. Use the following command when auto-enabling the S3 special remote:

git annex enableremote public-s3 \
publicurl="https://$BUCKET.s3-uhh.lzs.uni-hamburg.de"

🔗 Links

• https://git-annex.branchable.com/special_remotes/S3/
• https://git-annex.branchable.com/tips/public_Amazon_S3_remote/

Details about the object- / long-term storage (LZS)

Text copied from the LZS documentation on 10th of January (highlights added):

The RRZ operates a highly scalable object storage under the name Object / Long Term Storage (LZS). It complements the file service UHHDisk and the Central Storage Infrastructure ZSI to meet the constantly growing demand for storage space for data from science and research.

The LZS is suitable for, among others, the following purposes:

• Backend for file services
• Backend for archiving solutions
• Backend for research data management or CMS
• Backend for the delivery of multimedia files
• Storage of measurement data or results from simulation calculations
• Storage of larger collections of photos, films or digitized books
• On-premise cloud storage for scientific collaborations as a secure and affordable alternative to Amazon Simple Storage Service (S3)

The target group of the LZS are systems that speak S3 natively and have to store larger amounts of data at a low rate of data change. With UHHDisk, the RRZ offers a suitable platform for storing files, e.g. for home drives or project directories.

Due to the system architecture and configuration, the LZS is not suitable for use in structured data (backend for databases) or virtualization environments. For this the RRZ offers a suitable service with the central storage infrastructure (ZSI).

For further questions about the S3 object and long-term storage, please send an e-mail to the RRZ-ServiceLine.

Technical details of the LSZ

• Object Storage System: Cloudian HyperStore Object Storage
• Cloudian guarantees 99% compatibility to Amazon S3