Intro to Event Driven APIs with Amplify Streams & Twilio

This article explores Event Driven APIs & shows how I could transform a pull-style API such as NYT's one into a push-style one that sends headlines to my phone via SMS.

At least two reasons why you could need an Event-Driven API

Event Driven APIs allow applications to communicate together in a slightly different way than "traditional" APIs.

In a traditional REST API, for a simple use case such as retrieving data from a provider, the consumer App would have to query the provider through its API, generally sending HTTP requests to retrieve that data.

One of the problems that you may face as a developer of that consumer App is that you'd have to query & query over and over again to get "fresh" data. This means :

1/ generally taking the risk to over-consume the API in the sense that each API call is generally counted against a certain quota by the provider. And providers are doing it for good reasons because each request is loading their limited-resources servers. I'm talking about over-consuming because there's no guarantee that your next API calls will return any new data.

2/ If the provider includes indeed new data, let's say the latest 7 days weather forecast, your consumer app will have to handle sorting what's new vs what data you're already have. Not a big deal but not so fun to develop.

So we don't want to go back to the 90s and ask data providers to send nightly batches of data on which we will work the next day. We want to keep the data flow real time or near real time.

There comes event-driven APIs : the provider will still make his data available, but instead of expecting your consumer App to poll his data, he may present you directly or indirectly only the latest updates - ie the fresh incremental data only - but not only present it to you... but will send you events which include the data. That looks like a luxury API service isn't it ?

Yes it's so much a luxury that some of your favorite data providers will charge you for that : for example Twitter made it an Enterprise premium feature.

But what if you could transform any API as an event-driven stream ? You could be also in the data provider's seat and wanting people to have that luxury-style communication pattern (or more cheaply - UberEats style of data food delivered to your door when it's ready).

Streamifying the NYT API in a few clicks

In this example, I use Axway Amplify Streams to transform a regular API (the New York Times API, more specifically the News Wire API ) into a streaming, event-driven API. What I want is get notified each time the NYT sends a new article, by SMS directly to my phone. I want that notification to contain the headline. For the SMS part I use the Twilio API . Twilio could allow me to do much more, for example I could get those information sent to me directly via Whatsapp, with some media and links to the articles with a spefic query on a subject matter.

To start with, I need to make sure NYT will authenticate my app correctly. It is very straightforward. I create an App on their developer portal, and they provide me with an API Key & Secret:

Once done, I check from my browser that the API I want to use works well :

https://api.nytimes.com/svc/news/v3/content/all/all.json?api-key=[myAPIkeyhere]

This gives a bunch of articles as a result :

Now instead of jumping straight away to write my app against that API, I will go to Amplify Streams to transform it as an event-based API.

As you can see the service will "proxify" the polling process on my behalf, and from there I'll be able to send events to as many apps as I want. Visually & Conceptually Streams works like this .

I quickly test it to make sure it works, all directly from the portal :

I test the curl command from my local command line and observe a funny behaviour : I get my initial data (same data as I got from the browser), plus I see that Amplify Stream keeps sending a few updates from time to time. I'm on the right way :-)

Once this is done, I tweak my API configuration so that Streams authenticates well to the NYT API by putting the authentication info in the headers and not in the URL itself, and also define a reasonable polling frequency [disclaimer : I found this after a few errors when coding the nodejs app ] :

Now that my API is ready, and that I secured the communication between NYT API and Streams, I also want to make sure that my future app(s) will connect to my new event-driven API stream in a secure way. For that I go to Settings->Security and note down my App Token & private key in the signature section :

In order to finish my prep work, I also register for a trial account on Twilio's website. On a side note, I tried their serverless function feature in order to check that the SMS sending part worked and it's just amazingly simple, and super-well documented, kudos to their team. That accounts comes with a $15 credit, and you'll need to request an active phone number if you want to use the SMS API (it's the # section on the left menu). In the dashboard below, take note also of the Account SID & token :

A simple client App that will connect to NYT via Amplify Streams & notify me via Twilio

All I have to know is code a script locally - I could deploy it anywhere - that will be able to listen to events sent by Amplify Streams and each time send the title of the article to my phone. It can be in any language, I chose javascript on Node.js environment to show how this can run from any server-side environment. There are plenty of tutorials on how to install node.js, the main tasks I had to do on a Windows environment were : installing git, installing npm, and installing node.js with a few packages:

Now I was ready to install Amplify Streams SDK on my local server by running the following commands :

npm i streamdataio-js-sdk npm i streamdataio-js-sdk-auth

For the code I didn't need to start from scratch, by knowing little about node.js and not being a developer I could tweak the following script provided in the Github repo.

Here is the code I used :

let accountSid = '[provided by Twilio]'; //Twilio credentials
let authToken = '[provided by Twilio]';

const client = require('twilio')(accountSid, authToken);

var streamdataio = require('streamdataio-js-sdk/dist/bundles/streamdataio-node');
var AuthStrategy = require('streamdataio-js-sdk-auth');
var jsonPatch = require('fast-json-patch');
var print = require('node-print');

function server()
{
var targetUrl = 'https://api.nytimes.com/svc/news/v3/content/all/all.json';
  // appToken is the way Streamdata.io authenticates you as a valid user.
  // you MUST provide a valid token for your request to go through.
  var appToken = '[amplify streams app token]';
  var privateKey = '[amplify streams private key]';

  eventSource =
    streamdataio.createEventSource(targetUrl, appToken, [], AuthStrategy.newSignatureStrategy(appToken, privateKey));
  var result = [];

  eventSource
  // the standard 'open' callback will be called when connection is established with the server
    .onOpen(function ()
    {
      console.log("connected!");
    })
    // the streamdata.io specific 'data' event will be called when a fresh Json data set
    // is pushed by Streamdata.io coming from the API
    .onData(function (data)
    {
      console.log("initial data received from provider and being via SMS");
      // memorize the fresh data set
      result = data.results;   
        client.messages
          .create({
             body: result[0].title,
             from: '[phone number provided by Twilio]',
             to: '[my real phone number]'
           })
          .then(message => console.log(message));
      console.log(result[0].title + ' was sent');
    })
    // the streamdata.io specific 'patch' event will be called when a fresh Json patch
    // is pushed by streamdata.io from the API. This patch has to be applied to the
    // latest data set provided.
    .onPatch(function (patch)
    {
      // display the patch
      console.log("new data patch received from provider: ", patch);
      if(patch[0].op == 'add'){ //some updates from the NYT are just about publication dates and should be excluded
        console.log("new article detected...sending SMS");
        client.messages
          .create({
             body: patch[0].value.title,
             from: '+13133492283',
             to: '+336*********'
           })
          .then(message => console.log(message));
          console.log("sent " + patch[0].value.title + " to device");
      }
    })
    // the standard 'error' callback will be called when an error occur with the evenSource
    // for example with an invalid token provided
    .onError(function (error)
    {
      console.log('ERROR!', error);
      eventSource.close();

    });
  eventSource.open();
}
console.log('starting');
server();

To run the script I launch a Powershell command line, the console log was very helpful in debugging the issues:

That's it I receive a first sms when the script is launched and then a second one as soon as a new headline comes up

I hope you enjoyed the scenario. We only scratched the surface of what's being possible with Amplify Stream. In our example, we used standard HTTP Server-Sent Events to get the data you required through Amplify Streams proxy. SSEs are a W3C HTML5 standard for unidirectional real-time communication from a server to a client, where the connection keeps open and during which the server will push events to the client. This explains why my connexion kept open during my first curl command. Other subscription types are available in the product such as Webhooks & Kafka. Keep in mind that this is not an isolated service but the idea is that it is part of an API Management Suite with the synergies & capabilities someone would expect from such integration.

Thanks for reading me,

Happy integrations,

Walid Hajeri

Disclaimer : Views are my own , none of the ideas expressed in this post are shared, supported, or endorsed in any manner by my current employer.