How to use JZarr with AWS S3¶

In general JZarr can work with java.nio.file.Path objects. So if someone extends the abstract java.nio.file.FileSystem (see FileSystem) to connect an AWS S3 bucket this can be used to read from and write directly to such buckets.

In our example we use the Amazon-S3-FileSystem-NIO2 library which is forked several times by other implementors.

If you want to try the following example, add this maven dependency to your pom:

<dependency>
    <groupId>org.lasersonlab</groupId>
    <artifactId>s3fs</artifactId>
    <version>2.2.3</version>
</dependency>

Below you can see code snippets for connecting with, writing to and reading from an s3 bucket. You can find the entire example code here: S3Array_nio.java

connect an s3 bucket¶

Fill in your credentials.

code example for connecting the s3 bucket¶

String s3AccessKey = "<your access key>";
String s3SecretKey = "<your secret key>";
String s3Server = "s3.eu-central-1.amazonaws.com"; // example server name
String s3BucketName = "bucket-abcd"; // example bucket name

URI uri = URI.create(MessageFormat.format("s3://{0}:{1}@{2}", s3AccessKey, s3SecretKey, s3Server));
FileSystem s3fs = FileSystems.newFileSystem(uri, null);
Path bucketPath = s3fs.getPath("/" + s3BucketName);

write to an s3 bucket¶

In this example data will be written without compression. So you can easily check the chunk file content e.g. with an hex file viewer.

code example writing to an s3 bucket¶

Path groupPath = bucketPath.resolve("GroupName.zarr");
ZarrGroup group = ZarrGroup.create(groupPath);
ZarrArray array = group.createArray("AnArray", new ArrayParams()
        .shape(4, 8).chunks(2, 4).dataType(DataType.i1).compressor(CompressorFactory.nullCompressor));
byte[] data = {
        11, 12, 13, 14, 15, 16, 17, 18,
        21, 22, 23, 24, 25, 26, 27, 28,
        31, 32, 33, 34, 35, 36, 37, 38,
        41, 42, 43, 44, 45, 46, 47, 48
};
int[] shape = {4, 8};
int[] offset = {0, 0};
array.write(data, shape, offset);

read from an s3 bucket¶

code example reading from an s3 bucket¶

Path groupPath = bucketPath.resolve("GroupName.zarr");
final ZarrGroup group = ZarrGroup.open(groupPath);
ZarrArray array = group.openArray("AnArray");
byte[] bytes = (byte[]) array.read();
System.out.println(Arrays.toString(bytes));

The System.out should produce the following output:

[11, 12, 13, 14, 15, 16, 17, 18, 21, 22, 23, 24, 25, 26, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 41, 42, 43, 44, 45, 46, 47, 48]