How to use JZarr with AWS S3

In general JZarr can work with java.nio.file.Path objects. So if someone extends the abstract java.nio.file.FileSystem (see FileSystem) to connect an AWS S3 bucket this can be used to read from and write directly to such buckets.

In our example we use the Amazon-S3-FileSystem-NIO2 library which is forked several times by other implementors.

If you want to try the following example, add this maven dependency to your pom:

<dependency>
    <groupId>org.lasersonlab</groupId>
    <artifactId>s3fs</artifactId>
    <version>2.2.3</version>
</dependency>

Below you can see code snippets for connecting with, writing to and reading from an s3 bucket. You can find the entire example code here: S3Array_nio.java

connect an s3 bucket

Fill in your credentials.

String s3AccessKey = "<your acceyy key>";
String s3SecretKey = "<your secret key>";
String s3Server = "s3.eu-central-1.amazonaws.com"; // example server name
String s3BucketName = "bucket-abcd"; // example bucket name

URI uri = URI.create(MessageFormat.format("s3://{0}:{1}@{2}", s3AccessKey, s3SecretKey, s3Server));
FileSystem s3fs = FileSystems.newFileSystem(uri, null);
Path bucketPath = s3fs.getPath("/" + s3BucketName);

write to an s3 bucket

In this example data will be written without compression. So you can easily check the chunk file content e.g. with an hex file viewer.

Path groupPath = bucketPath.resolve("GroupName.zarr");
ZarrGroup group = ZarrGroup.create(groupPath);
ZarrArray array = group.createArray("AnArray", new ArrayParams()
        .shape(4, 8).chunks(2, 4).dataType(DataType.i1).compressor(CompressorFactory.nullCompressor));
byte[] data = {
        11, 12, 13, 14, 15, 16, 17, 18,
        21, 22, 23, 24, 25, 26, 27, 28,
        31, 32, 33, 34, 35, 36, 37, 38,
        41, 42, 43, 44, 45, 46, 47, 48
};
int[] shape = {4, 8};
int[] offset = {0, 0};
array.write(data, shape, offset);

read from an s3 bucket

Path groupPath = bucketPath.resolve("GroupName.zarr");
final ZarrGroup group = ZarrGroup.open(groupPath);
ZarrArray array = group.openArray("AnArray");
byte[] bytes = (byte[]) array.read();
System.out.println(Arrays.toString(bytes));

The System.out should produce the following output:

[11, 12, 13, 14, 15, 16, 17, 18, 21, 22, 23, 24, 25, 26, 27, 28, 31, 32, 33, 34, 35, 36, 37, 38, 41, 42, 43, 44, 45, 46, 47, 48]