S3PresignedPublisher class

The S3PresignedPublisher subclass of dataPublisher is for publishing data to S3 over a pre-signed URL.

Publishing data directly to S3 requires you to use a SigV4(a) signature in every request. Generating that URL signature is complex and requires communicating back and forth with temporary connections to get an access key and then using a series of calculations to get a final signing key. AWS support strongly recommends you don't try to generate this without one of their SDKs. There is a embedded C SDK for AWS with a SigV4 component library that can be used in embedded systems or with FreeRTOS, but there is not an Arduino port of it, nor do many Arduino boards meet the memory requirements for it.

If you cannot generate your own SigV4 signed request, you can use a pre-signed URL which is valid for a short time to access a single object (file) in a single S3 bucket. The pre-signed URL will give you the same permissions to the S3 object as the generator of the URL. There are a few caveats to using a pre-signed URL:

  • The URL is only valid for a single object (file). While you can reuse the URL, repeatedly using the same URL will over-write the same file it will not create a new one.
    • To write to a new file, you need a new URL.
  • The URL is only valid for a short time, usually only a few minutes.
    • The absolute maximum time a pre-signed URL could be valid for is 7 days, but they're frequently only valid for a few minutes or hours at most.

So, to publish to S3, this publisher provides a few options:

  • You can provide a set URL and set filename to use for the post. The set file will be read from the SD card. The exact URL will be used for the post. If you do not re-call the functions to update the URL and filename externally, the same S3 object will be over-written until the URL expires.
  • You can provide a set filename to use for the post and a function to call to get a new pre-signed URL. The set file will be read from the SD and the function will be called to generate a new URL based on that provided filename. If you do not call the function to update the filename externally, the same S3 object will be over-written with each post because the filename has not changed. But there should be no URL expiration because a new pre-signed URL will be generated each time.
  • You can provide a function to call to get a new pre-signed URL and no filename. A filename will be automatically generated based on the logger ID and the last marked date/time, assuming the file is of type #S3_DEFAULT_FILE_EXTENSION (".jpg"). The function to get a new pre-signed URL will be called based on the automatic filename. A new object will be created on S3 using the new url and logger/jpg based filename.
  • You can provide a function to call to get a new pre-signed URL and a file prefix and/or extension to use to generate a filename based on the date/time. The function to get a new pre-signed URL will be called based on the generated filename. A new object will be created on S3 using the new url and prefix/extension based filename.
  • You can provide a function to call to get a new pre-signed URL and a function to call to get a new filename. Both functions will be called and the upload to S3 will be based on the returned URL and filename.
    • NOTE: If your function to call to get the filename returns a nullptr, the default filename based on the logger ID, datetime, and #S3_DEFAULT_FILE_EXTENSION will be attempted.

The current tested path is:

  • The logger creates an image file on the SD card with the GeoLux camera (and also appends numeric data to a csv). The image file is named based on the logger ID and the marked date time from the sampling interval.
  • The logger calls the AWS IoT publisher
  • The AWS IoT publisher, while sending numeric data to AWS IoT Core, subscribes to its own S3 url topic.
  • The IoT publisher publishes a filename named based on the logger ID and the marked date time from the sampling interval to a basic-ingest topic tied to an IoT Core rule.
  • The AWS IoT Core rule triggers a lambda function to generate a pre-signed URL for the image file.
  • The lambda publishes the URL to the logger's pre-signed URL topic.
  • The subscribed IoT publisher receives the URL and passes it to the S3 publisher.
  • The logger calls the S3 publisher
  • The S3 publisher uses the logger to connect to the SD card and verify that it can open and read the GeoLux image file.
  • The S3 publisher uses the pre-signed URL to upload the GeoLux image file to S3.

Further documentation on how to set up the AWS IoT rule and lambda function and all of the proper permissions are forthcoming. Some documentation and an examples program using this library is available in the USGS NGWOS development repository.

Base classes

class dataPublisher
The dataPublisher class is a virtual class used by other publishers to distribute data online.

Constructors, destructors, conversion operators

S3PresignedPublisher()
Construct a new S3 Publisher object with no members set.
S3PresignedPublisher(Logger& baseLogger, const char* caCertName, String(*)(String) getUrlFxn = nullptr, String(*)(void) getFileNameFxn = nullptr, int sendEveryX = 1)
Construct a new S3 Publisher object.
S3PresignedPublisher(Logger& baseLogger, Client* inClient, String(*)(String) getUrlFxn = nullptr, String(*)(void) getFileNameFxn = nullptr, int sendEveryX = 1)
Construct a new S3 Publisher object.
~S3PresignedPublisher() virtual
Destroy the S3 Publisher object.

Public functions

void setHost(const char* host)
Set the S3 host name.
void setPort(int port)
Set the S3 port.
String getEndpoint(void) -> String override
Get the destination for published data - generally the host name of the data receiver.
void setURLUpdateFunction(String(*)(String) getUrlFxn)
Set function to use to get the new URL.
void setPreSignedURL(String s3Url)
Set the pre-signed S3 url.
void setFileName(String filename)
Set the filename to upload. If the filename is set, this exact filename will be used for the upload.
void setFileParams(const char* extension, const char* filePrefix = nullptr)
Set the filename parameters to use to auto-generate the filename before every post based on the file extension, prefix, and the current date and time.
void setFileUpdateFunction(String(*)(void) getFileNameFxn)
Set function to use to get the new URL.
void setCACertName(const char* caCertName)
Set the name of your certificate authority certificate file.
void begin(Logger& baseLogger, Client* inClient, String(*)(String) getUrlFxn)
Begin the publisher - linking it to the client and logger.
void begin(Logger& baseLogger, String(*)(String) getUrlFxn, const char* caCertName = nullptr)
Begin the publisher - linking it to the logger but not attaching a client.
int16_t publishData(Client* outClient, bool forceFlush = false) -> int16_t override
Utilizes an attached modem to make a TCP connection to the S3 URL and then stream out a get request over that connection.

Protected functions

Client* createClient() -> Client* override
Use the connected base logger's logger modem and underlying TinyGSM instance to create a new client for the publisher.
void deleteClient(Client* client) override
Delete a created client. We need to pass this through to avoid a memory leak because we cannot delete from the pointer because the destructor for a client in the Arduino core isn't virtual.
bool validateS3URL(String& s3url, char* s3host, char* s3resource, char* content_type) -> bool
Parses the S3 URL to validate that it is an un-expired S3 URL and fills the provided buffers with the parsed host, resource name, and content type from the URL.

Protected static variables

static const char* contentLengthHeader
The content length header text.
static const char* contentTypeHeader
The content type header text.

Protected variables

const char* s3_parent_host
The host name.
int s3Port
The host port.
String(* _getUrlFxn
Private reference to function used fetch a new S3 URL.
String _PreSignedURL
A pointer to the current S3 pre-signed URL.
String _filename
The name of the file you want to upload to S3.
const char* _filePrefix
The prefix to add to files, if generating a filename based on the date/time.
const char* _fileExtension
The extension to add to files, if generating a filename based on the date/time.
String(* _getFileNameFxn
Private reference to function used fetch a new file name.
const char* _caCertName
The name of your certificate authority certificate file.
File putFile
An internal reference to an SdFat file instance.

Function documentation

S3PresignedPublisher::S3PresignedPublisher(Logger& baseLogger, const char* caCertName, String(*)(String) getUrlFxn = nullptr, String(*)(void) getFileNameFxn = nullptr, int sendEveryX = 1)

Construct a new S3 Publisher object.

Parameters
baseLogger The logger supplying the data to be published
caCertName The name of your certificate authority certificate file - used to validate the server's certificate when connecting to S3 with SSL.
getUrlFxn A function to call to get a new pre-signed URL
getFileNameFxn A function to call to get a new filename
sendEveryX Interval (in units of the logging interval) between attempted data transmissions. NOTE: not implemented by this publisher!

S3PresignedPublisher::S3PresignedPublisher(Logger& baseLogger, Client* inClient, String(*)(String) getUrlFxn = nullptr, String(*)(void) getFileNameFxn = nullptr, int sendEveryX = 1)

Construct a new S3 Publisher object.

Parameters
baseLogger The logger supplying the data to be published
inClient An Arduino client instance to use to print data to. Allows the use of any type of client and multiple clients tied to a single TinyGSM modem instance
getUrlFxn A function to call to get a new pre-signed URL
getFileNameFxn A function to call to get a new filename
sendEveryX Interval (in units of the logging interval) between attempted data transmissions. NOTE: not implemented by this publisher!

void S3PresignedPublisher::setHost(const char* host)

Set the S3 host name.

Parameters
host The host name to use for S3 connections

This is "s3.<your-region>.amazonaws.com" by default. If you need to use a host in a specific region (ie, anything but US-East-1) you should set your own host. The host in that case should be: "s3.<your-region>.amazonaws.com"


void S3PresignedPublisher::setPort(int port)

Set the S3 port.

Parameters
port The port number to use for S3 connections

This is 443 by default. You only need to use this if you need a different port.


String S3PresignedPublisher::getEndpoint(void) override

Get the destination for published data - generally the host name of the data receiver.

Returns String The URL or HOST to receive published data

void S3PresignedPublisher::setURLUpdateFunction(String(*)(String) getUrlFxn)

Set function to use to get the new URL.

Parameters
getUrlFxn A function to call to get a new pre-signed URL

void S3PresignedPublisher::setPreSignedURL(String s3Url)

Set the pre-signed S3 url.

Parameters
s3Url The pre-signed URL to use to put into an S3 bucket

void S3PresignedPublisher::setFileName(String filename)

Set the filename to upload. If the filename is set, this exact filename will be used for the upload.

Parameters
filename The name of the file to be uploaded

void S3PresignedPublisher::setFileParams(const char* extension, const char* filePrefix = nullptr)

Set the filename parameters to use to auto-generate the filename before every post based on the file extension, prefix, and the current date and time.

Parameters
extension The file extension to use
filePrefix The prefix to use for the file name, optional, with an default value of nullptr. If not provided, the logger ID will be used.

void S3PresignedPublisher::setFileUpdateFunction(String(*)(void) getFileNameFxn)

Set function to use to get the new URL.

Parameters
getFileNameFxn A function to call to get a new filename

void S3PresignedPublisher::setCACertName(const char* caCertName)

Set the name of your certificate authority certificate file.

Parameters
caCertName The name of your certificate authority certificate file.

You MUST have already uploaded your certificate to your modem. This will most likely be the Amazon Root CA 1 (RSA 2048 bit key) certificate. You can find Amazon's current CA certificates here: https://docs.aws.amazon.com/iot/latest/developerguide/server-authentication.html. Depending on your module, you may instead need a certificate chain file or to use Amazon's older top-chain certificate (Starfield Services Root Certificate Authority - G2).

This is exactly the same CA certificate as you would use for an MQTT connection to AWS IoT (ie, the AWS IoT Publisher). For supported modules you can use the AWS_IOT_SetCertificates sketch in the extras folder to upload your certificate.


void S3PresignedPublisher::begin(Logger& baseLogger, Client* inClient, String(*)(String) getUrlFxn)

Begin the publisher - linking it to the client and logger.

Parameters
baseLogger The logger supplying the data to be published
inClient An Arduino client instance to use to print data to. Allows the use of any type of client and multiple clients tied to a single TinyGSM modem instance
getUrlFxn A function to call to get a new pre-signed URL

This can be used as an alternative to adding the logger and client in the constructor.


void S3PresignedPublisher::begin(Logger& baseLogger, String(*)(String) getUrlFxn, const char* caCertName = nullptr)

Begin the publisher - linking it to the logger but not attaching a client.

Parameters
baseLogger The logger supplying the data to be published
getUrlFxn A function to call to get a new pre-signed URL
caCertName The name of your certificate authority certificate file

This can be used as an alternative to adding the logger and client in the constructor.


int16_t S3PresignedPublisher::publishData(Client* outClient, bool forceFlush = false) override

Utilizes an attached modem to make a TCP connection to the S3 URL and then stream out a get request over that connection.

Parameters
outClient An Arduino client instance to use to print data to. Allows the use of any type of client and multiple clients tied to a single TinyGSM modem instance
forceFlush Ask the publisher to flush buffered data immediately.
Returns int16_t The http status code of the response.

This depends on an internet connection already having been made and a client being available.


Client* S3PresignedPublisher::createClient() override protected

Use the connected base logger's logger modem and underlying TinyGSM instance to create a new client for the publisher.

Returns Client* A pointer to an Arduino client instance

void S3PresignedPublisher::deleteClient(Client* client) override protected

Delete a created client. We need to pass this through to avoid a memory leak because we cannot delete from the pointer because the destructor for a client in the Arduino core isn't virtual.

Parameters
client The client to delete

bool S3PresignedPublisher::validateS3URL(String& s3url, char* s3host, char* s3resource, char* content_type) protected

Parses the S3 URL to validate that it is an un-expired S3 URL and fills the provided buffers with the parsed host, resource name, and content type from the URL.

Parameters
s3url The full S3 URL to parse
s3host A buffer for the parsed host name. Must be at least 95 characters long.
s3resource A buffer for the parsed resource. Should be long enough for the whole URL less the host.
content_type A buffer for the parsed and de-escaped content type. Must be at least 128 characters long
Returns bool True if the URL is valid and the provided buffers have been filled.

Do not use the values in the buffers before checking the return type!



Variable documentation

String S3PresignedPublisher::_PreSignedURL protected

A pointer to the current S3 pre-signed URL.

This must be a "virtual host style" URL. Path style URLs are not supported.


const char* S3PresignedPublisher::_filePrefix protected

The prefix to add to files, if generating a filename based on the date/time.


const char* S3PresignedPublisher::_fileExtension protected

The extension to add to files, if generating a filename based on the date/time.