A few weeks ago I had the opportunity to speak at an engineering meetup where I talked about serverless functions and demonstrated it in action using OpenFaaS. The live demo went fine without a glitch (thanks to the demo gods) so I thought it would be a good idea to write a blog post about it so here it goes.

Architecture Patterns

Monolithic architecture is the traditional model of application design. It is usually composed of components that are interconnected and interdependent with each other. The tightly-coupled nature of this architecture prevents teams from working independently which makes changes (especially in production) much more difficult.

Microservices is an architectural style that structures an application as loosely-coupled, highly maintainable, testable and independently deployable. The service is typically organized around business capabilities or functions (think Inverse Conway Maneuver). The modular design facilitates continuous delivery/deployment and enables the evolution of the technology stack.

Serverless on the other hand are "application designs that incorporate third-party “Backend as a Service” (BaaS) services, and/or that include custom code run in managed, ephemeral containers on a “Functions as a Service” (FaaS) platform". Serverless Functions are applications that are event-driven, ephemeral and stateless. It is on the extreme end of the microservices spectrum where it is further broken down into functions which typically follows the Unix philosophy of doing one thing and only one thing really well.

Benefits of Serverless Functions

Serverless Functions can be thought of as a form of utility computing as developers don’t have to care about the underlying infrastructure anymore because the details of provisioning and management has been abstracted away from them.

This is a boon as it allows a development team to focus on writing software that delivers business value rather than worry about all the surrounding plumbing such as environments, runtime, scaling, and monitoring.

The common use cases for using serverless functions are webhooks, machine learning, IoT and file processing.

Serverless Functions, Made Simple with OpenFaaS

There's a myriad of options available from the CNCF's serverless landscape with the big 3 cloud providers as the major players bringing their own offerings: Amazon Lambda, Microsoft Azure Functions and Google Cloud Functions.

Utilising the public cloud however is not always straight forward especially if you don't want to burn a hole in your pocket for conducting experiments and keeping up with the emerging technologies.

This is where OpenFaaS comes into play.

OpenFaaS® (Functions as a Service) is a framework for building Serverless functions with Docker and Kubernetes which has first-class support for metrics. Any process can be packaged as a function enabling you to consume a range of web events without repetitive boiler-plate coding.

OpenFaaS makes it easy to create your own FaaS platform and run serverless functions.

To demonstrate this, I will provide a walkthrough on two of the common use cases for serverless functions: webhooks and file/image processing.

Prerequisites

You need the following prerequisites in order to follow along:

• OpenFaaS deployed
• faas-cli installed
• Docker Hub ID
• MinIO installed and started
• MinIO client (mc) installed

Once you got these are sorted out, let's give this a spin.

Use Case: Image Processing

Let's create a decolorise function that converts a colored image to black and white.

1. Create the function scaffolding.

   From the shell prompt, run the following command (replace the --prefix value with your own Docker hub ID).

   $ faas-cli new --lang dockerfile decolorise --prefix riverron
   2019/06/10 22:50:25 No templates found in current directory.
   2019/06/10 22:50:25 Attempting to expand templates from https://github.com/openfaas/templates.git
   2019/06/10 22:50:28 Fetched 16 template(s) : [csharp csharp-armhf dockerfile dockerfile-armhf go go-armhf java8 node node-arm64 node-armhf php7 python python-armhf python3 python3-armhf ruby] from https://github.com/openfaas/templates.git
   Folder: decolorise created.
   ___                   _____           ____
   / _ \ _ __   ___ _ __ |  ___|_ _  __ _/ ___|
   | | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
   | |_| | |_) |  __/ | | |  _| (_| | (_| |___) |
   \___/| .__/ \___|_| |_|_|  \__,_|\__,_|____/
       |_|
    
    
   Function created in folder: decolorise
   Stack file written: decolorise.yml
   

   The resulting directory tree will look like this:

   $ tree
   .
   ├── decolorise
   │   └── Dockerfile
   └── decolorise.yml
   

2. Update the Docker image.

   We will be using imagemagick for doing the actual conversion so let's add it into the image.

   Using your favorite editor, edit the Dockerfile and add the following instruction:

   RUN apk --no-cache add imagemagick
   

   Save it and exit then run the following from the command line:

   $ perl -pi -e 's/fprocess="cat"/fprocess="convert - -colorspace Gray fd:1"/' decolorise/Dockerfile
   

3. Build/Push/Deploy the function.

   Note: Remember to docker login first prior to running the next command.

   $ faas-cli up -f decolorise.yml
   

4. Check function status and test it out.

   Once it is deployed, we can check the status by running:

   $ kubectl -n openfaas-fn get pods
   NAME                          READY   STATUS    RESTARTS   AGE
   decolorise-6dd4bd8d78-ccvwz   1/1     Running   0          11h
   $ faas-cli list
   Function                      	Invocations    	Replicas
   decolorise                    	0              	1
   

   Let's see if it works as expected by running:

   $ curl -sSL "https://images.pexels.com/photos/464376/pexels-photo-464376.jpeg?w=1260&h=750&auto=compress&cs=tinysrgb" | \
       faas-cli invoke decolorise > whale-bw.jpg
   

   You should see the original image converted to black and white:

   

Now that we have shown how we can use a serverless function for image processing, let's head over to the next use case.

Use Case: Webhooks

Another use case for using serverless functions is creating webhooks which are HTTP endpoints that accept a JSON payload. This payload is sent via POST method and is usually triggered as a result of an event.

Here's our objective for this use case:

• Create eventlistener function that will serve as a webhook.
• Configure MinIO to publish events via the eventlistener webhook.
• Add bucket notification when an image is uploaded.

Create eventlistener function.

1. Create the function scaffolding.

   $ faas-cli new --lang python3 eventlistener --prefix riverron
   

   The resulting scaffolding will look like:

   $ tree
   .
   ├── eventlistener
   │   ├── __init__.py
   │   ├── handler.py
   │   └── requirements.txt
   ├── eventlistener.yml
   

2. Create function handler and requirements file.

   eventlistener/handler.py

   from minio import Minio
   import requests
   import json
   import os
   import uuid
    
   def handle(req):
       """handle a request to the function
       Args:
           req (str): request body
       """
       payload = json.loads(req)
    
       mc = Minio(os.environ['minio_hostname'],
                   access_key=get_secret('access-key'),
                   secret_key=get_secret('secret-key'),
                   secure=False)
    
       records       = payload['Records'][0]['s3']
       source_bucket = records['bucket']['name']
       file_name     = records['object']['key']
       dest_bucket   = "images-processed"
    
       decolorise(source_bucket, dest_bucket, file_name, mc)
    
   def get_secret(key):
       val = ""
       with open("/var/openfaas/secrets/" + key) as f:
           val = f.read()
       return val
    
   def decolorise(source_bucket, dest_bucket, file_name, mc):
       mc.fget_object(source_bucket, file_name, "/tmp/" + file_name)
    
       f = open("/tmp/" + file_name, "rb")
       input_image = f.read()
    
       # convert image to black and white
       r = requests.post("http://gateway.openfaas:8080/function/decolorise", input_image)
    
       # write to temporary file
       temp_file_name = get_temp_file()
       f = open("/tmp/" + temp_file_name, "wb")
       f.write(r.content)
       f.close()
    
       # save processed image to Minio
       mc.fput_object(dest_bucket, file_name, "/tmp/" + temp_file_name)
    
       return file_name
    
   def get_temp_file():
       uuid_value = str(uuid.uuid4())
       return uuid_value
   

   eventlistener/requirements.txt

   minio
   requests
   

3. Update stack definition with MinIO hostname and secrets.

   In order for our function to talk to our object storage, we need to set the MinIO hostname and the corresponding secrets. These values are available during your initial MinIO setup.

   Let's create the secrets first.

   $ export ACCESS_KEY="<your_minio_access_key>"
   $ export SECRET_KEY="<your_minio_secret_key>"
   $ echo -n "${ACCESS_KEY}" | faas-cli secret create access-key
   $ echo -n "${SECRET_KEY}" | faas-cli secret create secret-key
   

   Then update the stack definition with the environment variable for the MinIO hostname and secrets.

   The final eventlistener.yml will look like this:

   provider:
   name: faas
   gateway: http://127.0.0.1:31112
   functions:
   eventlistener:
     lang: python3
     handler: ./eventlistener
     image: riverron/eventlistener:latest
     environment:
       minio_hostname: '<your_minio_hostname>:9000'
       write_debug: true
     secrets:
       - access-key
       - secret-key
   

4. Build/Push/Deploy the function.

   $ faas-cli up -f eventlistener.yml
   

5. Verify function status.

   $ kubectl -n openfaas-fn get pods
   NAME                             READY   STATUS    RESTARTS   AGE
   decolorise-6dd4bd8d78-ccvwz      1/1     Running   0          11h
   eventlistener-556d47ccbc-gw6lw   1/1     Running   0          32s
   $ faas-cli list
   Function                      	Invocations    	Replicas
   decolorise                    	7              	1
   eventlistener                 	0              	1
   

The function looks good but we're not done yet. Let's configure the MinIO side of things which we'll do next.

Configure MinIO to publish events via the webhook.

Following this instruction from the MinIO website, let's configure it to publish events to our eventlistener function:

1. Get the current config.

   $ mc admin config get myminio/ > /tmp/myconfig
   

2. Edit the saved config file.

   Look for the webhook section in /tmp/myconfig and set the endpoint to:

   http://127.0.0.1:31112/function/eventlistener
   

3. Deploy config with the updated webhook endpoint.

   $ mc admin config set myminio < /tmp/myconfig
   

4. Restart the MinIO server for the changes to take effect.

   $ mc admin service restart myminio
   

Add bucket notification for file uploads.

Using the MinIO client we will enable event notification to trigger our eventlistener function whenever a JPEG image is uploaded to the images bucket on myminio server.

$ mc mb myminio/images
$ mc mb myminio/images-processed
$ mc event add myminio/images arn:minio:sqs::1:webhook --event put --suffix .jpg

To check if event notification is configured successfully:

$ mc event list myminio/images

Putting it all together

So now we have the following stack:

1. decolorise function to convert images to black and white.
2. eventlistener function that invokes the decolorise function.
3. an event trigger configured on our object storage that POSTs to our eventlistener webhook.

Let's put this to the test by uploading a JPEG image to the bucket using the MinIO client:

$ mc cp /Users/riverron/Documents/engineeringmeetup/whale.jpg myminio/images
...meetup/whale.jpg:  120.26 KiB / 120.26 KiB  ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  100.00% 112.28 KiB/s 1s

The above action should have triggered the eventlistener webhook which then invoked the decolorise function and saved the converted image to the images-processed bucket.

Let's list the bucket contents to confirm:

$ mc ls myminio/images-processed
[2019-05-18 21:00:20 AEST] 145KiB whale.jpg

Voila! It worked!

Wrapping Up

In this post, I talked about the common use cases for using serverless functions and how simple it is to create using OpenFaaS. We created a file processing function and another function that serves as a webhook endpoint. Having the ability to run our own FaaS platform minimizes the barrier to entry, giving us the capability to evaluate emerging technologies and their applicable use cases.

Acknowledgments

This is based on various tutorials available in the community particularly those written by OpenFaaS founder Alex Ellis.