-->

dev openai, dalle, sms, aspnetcore, twilio

This article was originally published on the Twilio Blog.

Recently, there has been a massive boost in AI-generated art. It came to a point that an AI-generated piece of art won a contest a few months ago. There are many art generation programs available such as OpenAI DALL·E 2, Midjourney, Stable Diffusion, etc. In this article, you will use DALL·E 2 to generate images. They recently made their system available to the general public without a waitlist and also opened their API. They also give free credits so you can follow this article for free. You can also get the final source code from my GitHub repository.

Prerequisites

You’ll need the following things in this tutorial:

OpenAI and DALL-E 2

OpenAI started as a non-profit artificial intelligence research organization founded by Elon Musk and Sam Altman. Elon Musk later quit the company. Currently, it operates under OpenAI LP, a “capped-profit” company (a hybrid of profit and non-profit models).

DALL-E, is a machine learning model that uses GPT-3 to generate realistic images from a description. It was initially announced in January 2021. The latest iteration of the system, DALL-E 2, was announced in April 2022. It initially required joining a waiting list, and after you’ve been accepted, you could only generate images using their web front-end. Those limitations have now been lifted, and you can sign up and start using their API to generate images.

Overview of DALL-E 2 Front-End

When you go to the DALL-E 2 website and log in, you see an input box and a Generate button.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 1

The simplicity in the design reminds me of the Google homepage. It even has a “Surprise me” option which is similar to the “I’m feeling lucky” button.

Enter your description and press the Generate button. In a matter of seconds, you will see 4 image suggestions generated for you based on your description as shown below.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 2

You get 50 credits upon sign up but they expire after a month. After that, you get 15 free credits every month. 1 image generation costs 1 credit. You can check your credit status by clicking on your profile image on the upper right-hand corner.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 3

If you hover over the images, you see a “…” button appear. Click on it and the “Quick Actions” menu opens.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 4

Here you can download the image or generate more variations based on this. The new variations are quite similar to the original one though, as they are all based on the same description.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 5

You can keep on generating variations from variations as well. Generating variations also costs 1 credit.

Overview of OpenAI Account

To be able to use the OpenAI API, you will need an API key and some credits. When you sign up, OpenAI gives you free credits. Note that this is different from the 50 credits DALL-E 2 gave.

To check your credit status, go to your account page.

You should see your usage breakdown and your credit status.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 6

You get quite a lot of credits ($18) considering 1 image generation costs $0.02. There are even cheaper options depending on the image size and generation model.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 7

You can check the pricing page for full details.

After you’ve confirmed you have free credits granted, click on API Keys on the left menu.

Here, click on the Create new secret key button.

As the prompt says, save your secret key somewhere safe as you’ll not have another chance to see it.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 8

Now that you’ve familiarized yourself with image generation, have your API key and credits available, move on to the next section to implement your own API to send the DALL-E-2-generated images.

Project Implementation

To be able to generate the images, you will need to receive the image description from the user via SMS. To achieve this, you will implement a web API that responds to Twilio SMS webhook requests.

Create the API by running the following commands in a terminal:

mkdir Dalle2ImageSmsApi
cd Dalle2ImageSmsApi
dotnet new webapi

Since you’re going to use Twilio, add Twilio .NET SDK and ASP.NET helper library via NuGet:

dotnet add package Twilio
dotnet add package Twilio.AspNet.Core

Open the project with your IDE.

Under the Controllers directory, create a new file called IncomingSmsController.cs and update its contents with the code below:

using Microsoft.AspNetCore.Mvc;
using Twilio.AspNet.Core;
using Twilio.TwiML;
using Twilio.TwiML.Messaging;
namespace Dalle2ImageSmsApi.Controllers;
[ApiController]
[Route("[controller]")]
public class IncomingSmsController : TwilioController
{
    [HttpPost]
    public async Task<TwiMLResult> Index()
    {
        var form = await Request.ReadFormAsync();
        var incomingText = form["Body"];
        var message = new Message();
        message.Body($"Here's the image for your query: {incomingText}");
        message.Media(new Uri("https://picsum.photos/1024/1024"));
        return new MessagingResponse()
            .Append(message)
            .ToTwiMLResult();
    }
}

The code above extracts the text message sent by the user (which is sent in the Body paramater field of the form encoded request body.).

This message will be used to generate the image. For now, just for testing purposes, you will ignore this message and return a random photo from an online service called picsum.photos which is a handy service to create random placeholder images. You can use this service to test and format your response messages without wasting your OpenAI credits.

Run the application by running the following command in the terminal window:

dotnet run

You should see your application running on your localhost.

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 9

You want Twilio to send webhook requests to your API, but currently Twilio cannot access your localhost. To fix this issue, you will tunnel your localhost to the internet with ngrok.

Copy the localhost URL, open another terminal and run the following command:

ngrok http { YOUR LOCALHOST URL }

Replace { YOUR LOCALHOST URL } with the value you copied from the other terminal (It would be http://localhost:5252 in this example)

You should see ngrok generate a random Forwarding URL and ngrok will now forward the traffic from this URL to your local API:

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 10

Now that you have a publicly accessible URL, you can tell Twilio where to send webhook requests.

Go to Twilio Console. Then go to Phone Numbers → Manage → Active Numbers and click on your number.

Scroll down to the messaging section. Select Webhook in the “A MESSAGE COMES IN” part and enter your ngrok URL followed by /IncomingSms as shown below:

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 11

Click Save.

Now that the environment is set up, send an SMS to your Twilio phone and see if you can receive a random image as response. If you can receive the image, it means it’s now time to generate images using the OpenAI API.

Implement OpenAI Client

The image generation API is still in beta as of this writing. Using image generation is quite straightforward. You send the description of the image, size, and the number of images you want to get. In this project, you will use an open-source client library which has recently been updated to support DALL-E.

Stop your application if it’s still running and run the following command in the terminal window:

dotnet add package Betalgo.OpenAI.GPT3

You will need your OpenAI API key to use the API that you created in the previous section. You will use .NET user secrets to store it. Run the following command to initialize the user secrets:

dotnet user-secrets init

Then, add the API key to the secrets by running the following command, replacing {YOUR OPENAI API KEY} with the actual API key value:

dotnet user-secrets set OpenAIServiceOptions:ApiKey {YOUR OPENAI API KEY}

Update Program.cs and add the highlighted lines:

```csharp hl_lines=”1 11” using OpenAI.GPT3.Extensions; var builder = WebApplication.CreateBuilder(args); // Add services to the container. builder.Services.AddControllers(); // Learn more about configuring Swagger/OpenAPI at https://aka.ms/aspnetcore/swashbuckle builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); builder.Services.AddOpenAIService();


Now you can inject the service to your `IncomingSmsController` controller as shown below. The highlighted lines are what’s new and updated.

```csharp hl_lines="2 3 14 16 17 18 19 20 21 22 23 24 25 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 56"
using Microsoft.AspNetCore.Mvc;
using OpenAI.GPT3.Interfaces;
using OpenAI.GPT3.ObjectModels.RequestModels;
using Twilio.AspNet.Core;
using Twilio.TwiML;
using Twilio.TwiML.Messaging;
namespace Dalle2ImageSmsApi.Controllers;
[ApiController]
[Route("[controller]")]
public class IncomingSmsController : TwilioController
{
    private readonly ILogger<IncomingSmsController> _logger;
    private readonly IOpenAIService _openAiService;
    public IncomingSmsController(
        ILogger<IncomingSmsController> logger, 
        IOpenAIService openAiService
    )
    {
        _logger = logger;
        _openAiService = openAiService;
    }
    [HttpPost]
    public async Task<TwiMLResult> Index()
    {
        var form = await Request.ReadFormAsync();
        var incomingText = form["Body"];
        var createImageRequest = new ImageCreateRequest
        {
            Size = "1024x1024",
            N = 1,
            Prompt = incomingText,
            ResponseFormat = "url"
        };
        var createImageResponse = await _openAiService.Image.CreateImage(createImageRequest);
        if(!createImageResponse.Successful)
        {
            var errorMessage = "An error occurred trying to create OpenAI image." +
                $" {createImageResponse.Error.Code}: {createImageResponse.Error.Message}.";
            _logger.LogError(errorMessage);
            return new MessagingResponse()
                .Message("An unexpected error occurred. Try again later.")
                .ToTwiMLResult();
        }
        var image = createImageResponse.Results.First();
        var message = new Message();
        message.Body($"Here's the image for your query: {incomingText}");
        message.Media(new Uri(image.Url));
        return new MessagingResponse()
            .Append(message)
            .ToTwiMLResult();
    }
}

The action now construct a CreateImageRequest object with size set to 1024x1024, the number of images requested set to 1, and passes in the image description received from the incoming text message. It also sets the response format to url as it will pass Twilio the URL of the image. Valid values for ResponseFormat are url and b64_json.

Run the application again. To test the implementation, send an SMS to your Twilio phone again. This time the image returned should match your description.

For example, I sent the following description: “a golden retriever puppy playing with a kitten”:

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 12

When I clicked the link, I got the following image:

Generate images with DALL·E 2 and Twilio SMS using ASP.NET Core - image 13

I’m happy with the puppy but the kitten seems a bit odd. Of course, the nice thing about it is, if you don’t like the result, you can always request more.

As an improvement, you can make the image size and the number of images customizable.

Conclusion

In this tutorial, you learned the basics of DALL-E 2 via using the front-end. Then implemented your own service to interact with OpenAI API. The service responds to Twilio SMS webhooks and uses DALL-E 2 to generate images based on the user’s description and sends the image back to the user.

The resulting image may or may not be satisfactory based on the description. These are the early days of AI-generated images and I’m sure they will keep on getting better and better.

If you’d like to keep learning, I recommend taking a look at these articles:

dev dotnet, aspnetcore, voice, sendgrid, twilio

This article was originally published on the Twilio Blog.

Whether you like it or not, phone calls are essential to our daily communications. However, sometimes nobody is available to take the call right there and then. Luckily, Twilio Programmable Voice lets you record voicemail so the caller can leave a message. But what if instead of having to call into a voicemail box, you could receive the voicemail and transcript in as an email instead? In this article, you will build a Twilio Voice app that sends voicemails and the call transcript to your email address using SendGrid.

Prerequisites

You’ll need the following things in this tutorial:

Project Overview

Before jumping into the code, let’s take a look at how the application will work.

Take a look at this diagram of the application flow:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 1

  • Someone calls your Twilio Phone Number. Twilio picks up the call and forwards the details via HTTP to your Web API.

  • Your Web API responds with TwiML instructions. These instructions tell Twilio what to do with the phone call. You’ll learn more about TwiML later. Your TwiML instructions tell Twilio to record a voicemail and to send the recording transcript back to your Web API.

  • The caller leaves a message which Twilio records and transcribes.

  • When Twilio is done transcribing the recording, Twilio sends the transcription via HTTP to your Web API.

  • Your Web API will download the voicemail audio file (as MP3) and then use SendGrid to send an email with the phone number of the caller, the transcript of the voicemail, and the voicemail audio file itself.

  • SendGrid will deliver the email to your email inbox.

When Twilio receives a phone call, Twilio will send the details as an HTTP request to a URL that you configure and expect instruction as an HTTP response. This concept is called a webhook and is commonly used across Twilio products.

However, Twilio can only send HTTP requests to publicly available URLs, and you’ll be developing your application locally. To solve this, you’ll tunnel your localhost publicly using the free ngrok service. This wouldn’t be necessary in production, but is necessary for Twilio to reach your locally running web application. More on this later!

To instruct Twilio what to do with the phone call, you need to respond to the webhook HTTP request using a specific set of instructions called the Twilio Markup Language, or TwiML for short. TwiML is a specific set of XML tags that you can use to tell Twilio how to respond to voice calls and text messages. In this application you will use these two TwiML verbs: <Say> and <Record>.

<Say> will convert text to speech and send the audio to the caller. <Record> will record the audio of the phone call which you will use to implement voicemail functionality. These TwiML verbs can also have attributes and nested noun-tags. To instruct Twilio to transcribe the recording and send the transcription to your web application, you’ll be using the transcribe and the transcribeCallback attribute. Using these TwiML verbs and attributes, you’ll generate TwiML that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<Response>
  <Say>Hello. I'm not available at the moment. Please leave a message after the beep.</Say>
  <Record timeout="10" transcribe="true" transcribeCallback="/TranscribeCallback"></Record>
</Response>

!!!info

When you pass in a relative URL to transcribeCallback, Twilio will resolve the relative URL relatively to the URL it sent the HTTP request to. When using an absolute URL, Twilio will resolve the URL relatively to the root path of the URL it sent the HTTP request to.

!!!

This is all the TwiML you’ll be using in this application, but I recommend learning more about TwiML for Voice in the docs, and specifically to look deeper into the Say-verb and the Record-verb.

Now that you understand how the application will work, let’s get started.

Create the ASP.NET Core Web API

!!!info

If you’d prefer to get the final project directly, you can get it from my GitHub repository, or follow the steps below to implement it yourself.

!!!

The first step is to create a new Web API project to handle the Twilio webhooks and send the emails. You can do this by opening a terminal and running these commands:

mkdir VoicemailForwarderWebApi
cd VoicemailForwarderWebApi
dotnet new webapi

Run the application to confirm everything is in good order:

dotnet run

Your output should look like this:

info: Microsoft.Hosting.Lifetime[14]
      Now listening on: https://localhost:7117
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://localhost:5162
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.

This project template comes with a WeatherForecast controller. Open a new browser tab and browse to your HTTP URL with the /WeatherForecast path (in this example: http://localhost:5162/WeatherForecast). You should see results like this:

[{"date":"2022-08-17T13:41:31.816102+01:00","temperatureC":18,"temperatureF":64,"summary":"Cool"},{"date":"2022-08-18T13:41:31.817042+01:00","temperatureC":52,"temperatureF":125,"summary":"Balmy"},{"date":"2022-08-19T13:41:31.817048+01:00","temperatureC":21,"temperatureF":69,"summary":"Mild"},{"date":"2022-08-20T13:41:31.81705+01:00","temperatureC":-10,"temperatureF":15,"summary":"Mild"},{"date":"2022-08-21T13:41:31.817051+01:00","temperatureC":-19,"temperatureF":-2,"summary":"Mild"}]

This setup works fine in your local environment, but for Twilio to be able to send HTTP requests to your endpoints, your API needs to be publicly accessible over the internet.

You can achieve that with ngrok, which tunnels public requests to your local machine.

Leave your .NET app running, then open a separate terminal and run ngrok with the following command:

ngrok http YOUR_HTTP_PORT

You should see some random URL generated for you which is forwarding to your local API:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 2

Now append the endpoint path /WeatherForecast to the Forwarding URL and open it in a browser tab.

!!!info

If this doesn’t work for you, comment out app.UseHttpsRedirection(); in Program.cs and restart the application hitting ctrl + c and running dotnet run again. Alternatively, you can start ngrok with the following command:

ngrok http https://localhost:YOUR_HTTPS_PORT --host-header="localhost:YOUR_HTTPS_PORT"

!!!

You may see a warning message from ngrok:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 3

Click Visit Site to proceed.

You should see similar results to before, but now via ngrok’s public forwarding URL. This means that your API is publicly accessible and Twilio can send HTTP requests to it.

Receive Incoming Calls

Twilio provides libraries to make it easier to build Twilio applications. You will use two of those in this tutorial: The Twilio .NET SDK and the helper library for ASP.NET. You’ll use the SDK to generate TwiML and the helper library to respond to webhook requests.

Back in the terminal where your app is running, stop the application using ctrl + c and add the SDK and helper library for ASP.NET Core via NuGet:

dotnet add package Twilio
dotnet add package Twilio.AspNet.Core

Open the project in your IDE and add a new file in the Controllers folder called IncomingCallController.cs. Update the controller with the code below:

using Microsoft.AspNetCore.Mvc;
using Twilio.AspNet.Core;
using Twilio.TwiML;
namespace VoicemailForwarderWebApi.Controllers;
[ApiController]
[Route("[controller]")]
public class IncomingCallController : TwilioController
{
    [HttpPost]
    public TwiMLResult Index()
    {
        var response = new VoiceResponse();
        response.Say("So far, so good!");
        return TwiML(response);
    }
}

When Twilio sends an HTTP POST request to /IncomingCall this action will generate TwiML including the Say-verb which will instruct Twilio to say “So far, so good!” to the caller.

For Twilio to know where to send webhook requests, you need to update the webhook settings on your Twilio Phone Number.

Go to the Twilio Console. Select your account, and then click Phone Numbers → Manage → Active Numbers on the left pane. (If Phone Numbers isn’t on the left pane, click on Explore Products and then on Phone Numbers.)

Click on the phone number you want to use for your project and scroll down to the Voice section.

Under the “A Call Comes In” label, set the dropdown to Webhook, the text field next to it to the ngrok Forwarding URL suffixed with the /IncomingCall path, the next dropdown to HTTP POST, and click Save. It should look like this:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 4

Now, run the application (dotnet run) and call your Twilio number, and you should hear the message “So far, so good” on your phone. Great job if this is working. If not, there are a couple of places where you can go to debug:

  • You may see errors in the output from your .NET application in the terminal

  • Check the output of the ngrok command in the other terminal, or browse to the ngrok dashboard (http://127.0.0.1:4040) where you can inspect HTTP requests and responses.

  • You can find errors and call details in the Twilio Console under the Monitor tab.

Now that you verified the webhook is working, let’s update the TwiML so the caller can leave a voicemail. To do this, update the Say-verb to prompt the user to leave a message, and use the Record-verb to record the call.

Update your Index method as below:

var response = new VoiceResponse();
response.Say("Hello. I'm not available at the moment. Please leave a message after the beep.");
response.Record(
    timeout: 10,
    transcribe: true,
    transcribeCallback: new Uri("/TranscribeCallback", UriKind.Absolute)
);
return TwiML(response);

The timeout attribute tells Twilio to end the recording after a number of seconds of silence has passed. The default is 5 seconds. You can change this to your liking.

By setting the transcribe attribute to true, you’ll instruct Twilio to transcribe the recording. Twilio will also store the transcription so you can retrieve the transcription later via the Twilio API.

The transcription process happens asynchronously. Twilio can send the transcript data to your application when it is ready. Use the transcribeCallback attribute to tell Twilio to which URL to send the transcription data when it is ready.

Now move on to handle the transcription webhook.

Receive Transcription Text and Recording Info

Twilio returns the transcription text and the recording URL in the transcribe callback message. So you can gather everything you need by handling the transcribe callback.

Add a new file under the Controllers folder named TranscribeCallbackController.cs. Update its contents as below:

using Microsoft.AspNetCore.Mvc;
using Twilio.AspNet.Core;
namespace VoicemailForwarderWebApi.Controllers;
[ApiController]
[Route("[controller]")]
public class TranscribeCallbackController : TwilioController
{
    private readonly ILogger<TranscribeCallbackController> _logger;
    public TranscribeCallbackController(ILogger<TranscribeCallbackController> logger)
    {
        _logger = logger;
    }
    [HttpPost]
    public async Task Index()
    {
        var form = await Request.ReadFormAsync();
        var recordingSid = form["RecordingSid"].ToString();
        var recordingUrl = form["RecordingUrl"].ToString();
        var transcriptionText = form["TranscriptionText"].ToString();
        var callingNumber = form["From"].ToString();
        _logger.LogInformation("Transcription details -> CallingNumber: [{callingNumber}] TranscriptionText: [{transcriptionText}], RecordingSid: [{recordingSid}], RecordingUrl: [{recordingUrl}]", 
            callingNumber, transcriptionText, recordingSid, recordingUrl);
    }
}

To test your changes, restart your .NET application, then call your Twilio number again and leave a message. After a few seconds, you should see a new log line like this in your terminal:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 5

!!!warning

By default, Recording URLs don’t require authentication, and recordings are not encrypted. However, you can require basic authentication to access the recordings and configure recordings to be encrypted in the voice settings (Voice → Settings → General).

!!!

Download the call recording

The next step is to get the recording audio. The audio is available in two formats: WAV and MP3. WAV files are uncompressed and have larger file sizes. Since they will be sent as attachments, in this example, you will download the MP3 version for efficiency.

First, you will need an HttpClient to download the file. Add the highlighted line below to your Program.cs file:

```csharp hl_lines=”3” builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); builder.Services.AddHttpClient();


Update TranscribeCallbackController.cs so that the constructor and the private variables look like below:

```csharp hl_lines="2 4 7"
private readonly ILogger<TranscribeCallbackController> _logger;
private readonly IHttpClientFactory _httpClientFactory;
public TranscribeCallbackController(ILogger<TranscribeCallbackController> logger, IHttpClientFactory httpClientFactory)
{
    _logger = logger;
    _httpClientFactory = httpClientFactory;
}

Now you can instantiate an HttpClient and download the file at recordingUrl by adding the following code to your Index() method in the TranscribeCallbackController:

var httpClient = _httpClientFactory.CreateClient();
var recordingBytes = await httpClient.GetByteArrayAsync($"{recordingUrl}.mp3");
var recordingFilePath = $"{recordingUrl.Substring(recordingUrl.LastIndexOf("/") + 1)}.mp3";
System.IO.File.WriteAllBytes(recordingFilePath, recordingBytes);

By default, the recording URL doesn’t have a file extension. If you call the URL as is, Twilio returns the WAV version of the recording. To get the MP3 version, you need to append the .mp3 extension to the URL as shown above.

Restart your application, call your Twilio number again and leave a message. Once Twilio sends the transcription to your application, you should see an MP3 file appear in your project folder. You won’t actually need to save the file to disk for this tutorial, but you can do this to quickly test that it works so far.

Send the Voicemail via Email

Now that you have the transcribed text and the call recording audio, the final step is to create an email and send these to your email address.

To achieve this, you’re going to use SendGrid SDK.

Stop the application and add the following packages via NuGet:

dotnet add package SendGrid
dotnet add package SendGrid.Extensions.DependencyInjection

To send emails via SendGrid, you will need to use your API key and store it somewhere. You can use environment variables or a vault service, but for local development you can use dotnet user secrets. First, you need to initialize user secrets by running

dotnet user-secrets init

Then, create a new user secret called SendGrid:ApiKey and set your API key:

dotnet user-secrets set SendGrid:ApiKey {YOUR SENDGRID API KEY}

Replace {YOUR SENDGRID API KEY} with your SendGrid API Key (see prerequisites).

Now apply the following code changes. First, update Program.cs as shown below:

```csharp hl_lines=”4” builder.Services.AddEndpointsApiExplorer(); builder.Services.AddSwaggerGen(); builder.Services.AddHttpClient(); builder.Services.AddSendGrid(options => options.ApiKey = builder.Configuration[“SendGrid:ApiKey”]);


And add the following using statement to the top of the Program.cs file:

```csharp
using SendGrid.Extensions.DependencyInjection;

Then, update your TranscribeCallbackController to inject the ISendGridClient and store it in a private field:

```csharp hl_lines=”3 8 13” private readonly ILogger _logger; private readonly IHttpClientFactory _httpClientFactory; private readonly ISendGridClient _sendGridClient; public TranscribeCallbackController( ILogger logger, IHttpClientFactory httpClientFactory, ISendGridClient sendGridClient ) { _logger = logger; _httpClientFactory = httpClientFactory; _sendGridClient = sendGridClient; }


Also, add these using statements to the top of the TranscribeCallbackController.cs file:

```csharp
using SendGrid;
using SendGrid.Helpers.Mail;

Finally, update the Index method of the TranscribeCallbackController controller so that the final version looks like this:

var form = await Request.ReadFormAsync();
var recordingSid = form["RecordingSid"].ToString();
var recordingUrl = form["RecordingUrl"].ToString();
var transcriptionText = form["TranscriptionText"].ToString();
var callingNumber = form["From"].ToString();
_logger.LogInformation("Transcription details -> CallingNumber: [{callingNumber}] TranscriptionText: [{transcriptionText}], RecordingSid: [{recordingSid}], RecordingUrl: [{recordingUrl}]", 
    callingNumber, transcriptionText, recordingSid, recordingUrl);
var httpClient = _httpClientFactory.CreateClient();
var recordingBytes = await httpClient.GetByteArrayAsync($"{recordingUrl}.mp3");
var from = new EmailAddress("{your sender email}", "{your sender display name}");
var to = new EmailAddress("{your recipient email}", "{your recipient display name}");
var subject = "You've got voicemail!";
var plainTextContent = $"Calling Number: {callingNumber}{Environment.NewLine}Transcription: {transcriptionText}";
var htmlContent = $"<p>Calling Number: {callingNumber}</p><p>Transcription: {transcriptionText}</p>";
var msg = MailHelper.CreateSingleEmail(from, to, subject, plainTextContent, htmlContent);
msg.AddAttachment(
    new Attachment
    {
        Content = Convert.ToBase64String(recordingBytes),
        Filename = "voicemail.mp3",
        Type = "audio/mpeg",
        Disposition = "attachment"
    });
var sendEmailResponse = await _sendGridClient.SendEmailAsync(msg);
_logger.LogInformation(sendEmailResponse.IsSuccessStatusCode ? "Email queued successfully!" : "Something went wrong!");

!!!warning

Since you don’t know what’s in the recording and the transcription text, you should assume it could contain personal information. I am logging the transcription text for debugging purposes, but you should avoid doing so in production to protect PII.

!!!

Before running the application, replace {your verified sender email}, {your sender display name}, {your recipient email} and {your recipient display name} with actual values. Display names can be anything you choose. The sender email address needs to be a verified sender in SendGrid (see prerequisites).

Start your application again as before:

dotnet run

The code above creates a new email with the MP3 file as an attachment, and the phone number and transcribed text as the body of the email.

Test your application one last time by calling your Twilio number and leaving a voice message. After a few seconds, you should receive an email that looks like this:

Forward Voicemails with Transcript to your Email using C# and ASP.NET Core - image 6

Conclusion

The email you send may not look pretty, but it does the job. You can use SendGrid Dynamic Email Templates and create beautiful HTML email templates. If you are interested in sending templated emails with SendGrid, take a look at these articles:

To find out more about using voicemails with Twilio, tunneling, and ngrok, here are some of the articles to read:

dev aws, lex, chatbot, sms, dotnet, twilio

This article was originally published on the Twilio Blog.

There is a tough competition in the business world to acquire and retain customers. One key step to achieve this goal is to keep your customers happy with your customer support. Having an automated chatbot helps your business provide a faster and more accessible customer support to your customers. In this article, you will learn how to build a chatbot using C#, AWS Lambda, Amazon Lex, Amazon DynamoDB, and Twilio SMS.

Prerequisites

You’ll need the following things in this tutorial:

What is Amazon Lex?

Amazon Lex is an artificial intelligence service that allows developers to create voice or text-based conversational interfaces. This service powers Amazon’s own Alexa.

Lex provides automatic speech recognition and natural language understanding technologies. It takes the user’s input, runs it through a Natural Language Processing (NLP) engine and determines the user’s intent. The value of this is the user does not need to remember a set of commands to interact with your bot. They can talk to the bot just like they would to a human being.

This project uses several AWS services: Lex, Lambda and DynamoDB. To follow along, you will need an IAM user setup in your development environment. Proceed to the next section for the IAM setup. If you already have it configured, you can skip the next section and move on to the Project Overview.

If you created a new AWS account, this project shouldn’t cost you anything, as all these services have free tiers. If you are on an older account, it shouldn’t cost too much. Still, I recommend checking the pricing pages of the services anyway: Amazon Lex Pricing, AWS Lambda Pricing and Amazon DynamoDB pricing.

Set up AWS IAM User

You will need credentials to deploy your application to AWS from the command line. To create the credentials, follow the steps below:

First, go to the AWS IAM Users Dashboard and click the Add users button.

Enter the user name, such as twilio-webhook-user and tick the Access key - Programmatic access checkbox:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 1

Click the Next: Permissions button at the bottom right.

Then, select Attach existing policies directly and select AdministratorAccess:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 2

Click the Next: Tags button at the bottom right. Tags are optional (and quite valuable information), and it’s a good practice to add descriptive tags to the resources you create. Since this is a demo project, you can skip this step and click the Next: Review button at the bottom.

Confirm your selection on the review page. It should look like this:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 3

Then, click the Create user button.

In the final step of the user creation process, you should see your credentials for the first and the last time.

!!!warning

Take note of your Access key ID and Secret access key before you press the close button.

!!!

Now, open a terminal window and run the following command:

aws configure

You should see a prompt for AWS Access Key ID. Copy and paste your access key ID and press enter.

Then, copy and paste your secret access key and press enter.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 4

When prompted, type us-east-1 as the default region name and press enter.

!!!info

In this example, I will use the us-east-1 region. Regions are geographical locations where AWS have their data centers. It is a good practice to deploy as close to your customers as possible for production deployments to reduce latency. Since this is a demo project, you can use us-east-1 for convenience as it’s the default region in AWS Management Console. You can find more on AWS regions in this document: Regions and Availability Zones.

!!!

As the default output format, type json and press enter.

To confirm you have configured your AWS profile correctly, run the following command:

aws configure list

The output should look like this:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 5

Now that you have set up your AWS credentials, you can move on to the demo project.

Project Overview

The application you will implement is an imaginary online stock broker customer service. It will accept requests like buy, sell, show portfolio, etc.

Take a look at this diagram of the application flow:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 6

  • Customer sends an SMS to customer service (which is a Twilio Phone Number).

  • Twilio receives the message and invokes the Amazon Lex callback.

  • Amazon Lex identifies the user’s intent and calls the corresponding Lambda function.

  • The Lambda function executes the application logic based on the request, gets the customer info from the DynamoDB database, and prepares the response.

  • Lex sends the response to Twilio using the Twilio SMS integration.

  • Twilio delivers the response message to the customer’s phone.

Without further ado, let’s get the demo application and start exploring the existing code.

Set up the Demo Project

The focus of this article is developing a chatbot using Amazon Lex and Twilio SMS. To save time, the fundamental business logic of the fictional stock broker is implemented in the starter project.

Clone the project to get started:

git clone https://github.com/cloudinternals/amazon-lex-stock-broker-bot-with-twilio-sms.git --branch starter-project

Open the solution (src/StockBrokerBot/StockBrokerBot.sln) in your IDE and take a look at the project structure. The StockBrokerBot.Core project looks like this:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 7

The main functionality is in 2 services: IPortfolioService and IStockMarketService.

IPortfolioService shows the behaviour of the service:

public interface IPortfolioService
{
    Task<UserPortfolio> GetUserPortfolio(string userId);
    Task<UserPortfolio> BuyStocks(string userId, string stockName, decimal numberOfShares);
    Task<UserPortfolio> SellStocks(string userId, string stockName, decimal numberOfShares);
}

It supports 3 operations: Get portfolio, buy stocks, and sell stocks.

The core library includes one implementation of the portfolio service called PortfolioService. It depends on a stock market service and a portfolio data provider:

private readonly IStockMarketService _stockMarketService;
private readonly IPortfolioDataProvider _dataProvider;
public PortfolioService(IStockMarketService stockMarketService, IPortfolioDataProvider dataProvider)
{
    _stockMarketService = stockMarketService;
    _dataProvider = dataProvider;
}

PortfolioService performs validations during buy and sell operations (user has sufficient funds to buy, or shares available to sell, etc.)

IStockMarketService is responsible for fetching the current stock price:

public interface IStockMarketService
{
    Task<decimal> GetStockPrice(string stockName);
}

There are 2 different stock market service implementations: StockMarketService, and FluctuatingStockMarketService.

StockMarketService simply fetches the stock price from its data provider. FluctuatingStockMarketService is meant to “spice things up” a little bit. It calculates a random price by adding a small price swing within 2%. You can, of course, change this rate to create higher swings. This way, you will get a new price every time. So you can buy low and sell high and make some imaginary profits!

The starter project also includes a demo console application. The demo application uses JSON files to persist user portfolios and stock prices. In the actual chatbot, you will use DynamoDB tables. The demo project uses the FluctuatingStockMarketService. You can replace it with StockMarketService to get more consistent results.

UserPortfolio and Stock entities are already annotated to be used as DynamoDB entities:

[DynamoDBTable("user-portfolio")]
public class UserPortfolio
{
    [DynamoDBHashKey]
    public string UserId { get; set; }

!!!info

In a real project, I wouldn’t recommend creating a dependency on a storage provider from your business library but for the sake of brevity, the same entities will be used in this project.

!!!

Open a terminal, navigate to the demo project, and run the application by running:

cd StockBrokerBot.Demo
dotnet run

You should see the results that look like this:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 8

Take some time to look into the core library and the consuming demo application. There is also a Lambda function project in the solution called StockBrokerBot.ChatbotLambda, but it’s empty at the moment. You will implement it while following this article.

When you are more familiar with the project, move on to the next section to create the chatbot.

Create the Chatbot with Amazon Lex

To start implementing your bot, go to Lex Console and click Create bot.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 9

In the settings, leave the Create a blank bot option selected.

In the Bot name field, enter StockBrokerBot.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 10

In IAM permissions, select Create a role with basic Amazon Lex permissions.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 11

Select No in the COPPA section and click Next.

In the Add languages step, leave the default language (English (US)). You can choose to support multiple languages and even assign a different voice to each language. In this article, you will use a single language and text interaction only. Click the Voice interaction dropdown, scroll to the bottom and select None. This is only a text-based application option.

Click Done to create your bot.

Your bot has now been created, and Lex redirects you to create your first intent. An intent is an action your bot takes to fulfil a user’s request.

In the Intent name field, enter CheckStockPrice.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 12

Leave Contexts blank and scroll to Sample utterances.

An utterance is a phrase that corresponds to this intent. In conversations, we use many different phrases to express the same thing. For example, if you have a Hello intent, sample utterances can be “Hello”, “Hi”, “Hey” etc.

Click Plain Text and paste the following utterances in the field:

check the {stockName} price
check {stockName}
how much is {stockName} ?
what is the price of {stockName} ?
get the price of {stockName}
{stockName}
price {stockName}
what is the current {stockName} price?
check the current price of {stockName}
price of {stockName} ?

In the above text block, you can see many occurrences of {stockName}. This is what is called a slot. It’s essentially a placeholder for a piece of data you need Lex to extract for you and pass it on to your code. If you recall the core library introduced earlier in the article, GetStockPrice method requires the stock name. A user might express their intention in a lot of different ways. Extracting this data is Lex’s responsibility so that, as the bot developer, you can focus on your bot’s business logic.

!!!warning

The space between the slot and the question mark at the end is intentional. Lex requires spaces surrounding the slots. If you remove those spaces, you will get an error while saving the intent.

!!!

Scroll down to the Slots section.

As discussed above, you’re using a slot in your utterances, but it’s not defined yet. Lex needs to know the type and whether or not it’s mandatory. Lex performs much better if you train what kind of data it’s looking for. In the stock name example, there is a finite set of company names, so you will create your own data type to train the model better. Skip adding a slot for now. You’ll revisit this part very soon.

Click the Save intent button and the Back to intents list link on the left pane. Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 13

!!!info

FallbackIntent is one of the built-in intents. If the user’s request doesn’t match any of the intents, FallbackIntent is invoked. You can read more about Lex’s built-in intents here.

!!!

On the left menu, click Slot types which is right under Intents.

Click Add slot type and select Add blank slot type.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 14

Enter StockName in the Slot type name field.

In the Slot value resolution, leave the Expand values option selected. You can also choose to restrict the slot values, but then you will need to enter every stock name that your service supports. It almost becomes a lookup table. Lex is smart enough to identify similar values based on your training set. The more comprehensive your training set is, the better results you will get.

Enter Apple, Alphabet, Microsoft, Tesla, and Twilio as stock names and click Save slot type.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 15

Click the Slot types link on the left, then Intents, and finally, click on the CheckStockPrice intent to get back to intent settings.

Scroll down to the Slots section and click Add slot.

Leave Required for this intent checkbox ticked.

Enter stockName in the Name field and select StockName in the Slot type list.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 16

In the Prompts field, enter What is the name of the stock?

This is a very useful feature. You don’t have to worry about asking the user for the stock name if it’s missing. Lex will automatically ask the user and fill in the missing values, so you can rest assured that it will always deliver the required values to your bot’s backend. You’ll see this in practice while testing the bot later on.

Click Add to close the dialog and then click Save Intent.

Now focus on the top of the screen.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 17

You should see the version you’re looking at (Draft version), the language (English (US)) and a label next to it that says “Not built”.

Click the Build button for Lex to build the machine learning (ML) model for your bot. You cannot test your bot without creating the ML model first. When the build is complete, you will see a notification on your screen.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 18

Click the Test button.

In the bottom field (with the Type a message placeholder), enter “What is the price of Apple?” and press enter.

You should see a message that says “Intent CheckStockPrice is fulfilled”

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 19

Click the Inspect button to see more details. You can see the stockName slot has Apple as the value. If you recall, you made the stockName a required slot. To test what happens if a user enters a message without providing sufficient information, enter “price” as the message and press enter.

!!!info

Note the title of the dialog says “Test Draft version”. When you test, you test the entire model for the selected language, not a single intent, even if you open the dialog while you are on an intent page.

!!!

You should see Lex now asks the name of the stock explicitly. The question it asks is the prompt message you entered when you created the slot type.

Just to emphasise how it works, it’s not directly looking up the utterances and matching strings to determine the intent. For example, you can express the same intent by entering “show the price of Tesla stock”, and you should still see the intent is fulfilled message even though it’s not part of the utterance list. If the expected intent is not fulfilled, you can modify your utterances, rebuild, and retest the model.

!!!warning

Before you test, make sure to build your model if you’ve made any changes. Otherwise, you’d be testing the previous model.

!!!

After the intent is recognized, what Lex will do with it is determined by the Fulfillment settings. By default, fulfilment is not active. Scroll down to the Fulfillment section and click the Active radio button. Lex invokes the Lambda function associated with your bot by default. You can confirm this behaviour by expanding the parameters and clicking the Advanced options button.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 20

Set the Fulfilment to active and ensure the Use a Lambda function for fulfilment option is ticked.

Click Save Intent and then click the Back to intents list link on the left.

Click the Add Intent button. It will show you two options: Add empty intent and Use built-in intent)

Select Add empty intent.

Enter GetPortfolio as intent name in the dialog and click Add.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 21

In the Sample utterances section, switch to Plain Text view and paste the following utterances:

​​get portfolio
show my portfolio
my portfolio
my stocks
show me the money!

In the Fulfillment section, set the Active option to true.

Click the Save intent button and Back to intents list link on the left.

Click Add intent again and set the intent name to BuyStocks. Update the utterances with the ones below as you did before:

buy {numberOfShares} of {stockName} 
buy shares {numberOfShares} ,stock {stockName}
buy {numberOfShares} {stockName}
purchase {numberOfShares} of {stockName} stock
get {numberOfShares} shares of {stockName}
buy {stockName} {numberOfShares}  shares
buy {numberOfShares} shares of {stockName}
buy {numberOfShares} shares of {stockName} stock

In the Slots section, click the Add slot button.

In the Add slot dialog, set Required for this intent to true, enter stockName as the name, select StockName as slot type and “What is the name of the stock?” as the prompt.

Click Add.

Click the Add slot button again.

This time, set the name to numberOfShares, slot type to AMAZON.Number and the prompt to “How many shares?”.

Click Add again to save the second slot.

In the Fulfilment section, set the Active option to true.

Click the Save intent button and Back to intents list link on the left.

Click Add intent for one last time and set the intent name to SellStocks. Update the utterances with the ones below as you did before:

sell {numberOfShares} shares of {stockName}
sell shares, number: {numberOfShares} , stock: {stockName}
sell {numberOfShares} of {stockName}

SellStocks intent is very similar to the BuyStocks intent. Create the same slot types as the BuyStocks intent as described above.

In the Fulfilment section, set the Active option to true.

Click the Save Intent button.

Now that all the intents have been described, click the Build button to rebuild the model.

After a show while, you should get a Successfully built notification:

Dismiss the notification and click the Bot: StockBrokerBot link in the breadcrumb.

In the left menu, under your bot there is a Bot versions link, and under it the Draft version which you’ve been working on.

Click Bot versions, and then click Create version.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 22

In the Description field, enter a description such as “Initial version with four intents” and click Create button at the bottom of the page.

You don’t assign version numbers, they are auto-incrementing integers. After you’ve created the version, it should appear in the version list:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 23

A version is essentially a read-only snapshot of your bot. You cannot modify a version after you’ve published it.

Now, take a look at another important concept: Aliases.

Click Aliases link on the left menu. It should show the default TestBotAlias:

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 24

!!!info

An alias is associated with a specific version of your bot. The benefit of this is you can have multiple aliases such as test and live. If you publish a new version, you can point the test alias to the new version. This way your live alias is not affected until you test your changes. After you’re satisfied your new version is ready to go live, you can simply associate the live alias with the new version and all the new requests will come to the new version of your bot. Also, if you experience issues with your latest version, you can simply assign the previous version to your alias to roll back. This kind of separation between the versions and aliases makes change management a lot easier.

!!!

Click the Create alias button.

Enter Live as Alias name.

In the Associate with a version section, choose Version 1. The language comes already enabled so leave it like that.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 25

Click the Create button.

You should see the alias is successfully created and shown in the list:

Now Version 1 of your bot has been published.

You will assign a Lambda function to your bot, but first, move on to the next section to create the backend of your bot.

Create the Backend

Open a terminal and navigate to the directory that will be the root of your project.

You will need the Amazon Lambda Tools .NET tool to deploy the function via the command line. You can install it by running the command below:

dotnet tool install -g Amazon.Lambda.Tools

As shown in the demo project, you will need two data sources: one to store users’ portfolios and the other to store stock prices. (The following scripts can be found in the Setup/InfrastructureSetup.sh file in the ChatbotLambda project.)

To create the user portfolio table, run the following command:

aws dynamodb create-table \
    --table-name user-portfolio \
    --attribute-definitions \
        AttributeName=UserId,AttributeType=S \
    --key-schema \
        AttributeName=UserId,KeyType=HASH \
    --provisioned-throughput \
        ReadCapacityUnits=5,WriteCapacityUnits=5 \
    --table-class STANDARD

The script above creates a DynamoDB table with the UserId partition key, which you will use to query and fetch the users’ records.

To keep things simple, the account creation process is omitted. To create the account, add your user’s portfolio directly to the database by running the following command (replace { YOUR PHONE NUMBER WITH COUNTRY CODE } with your actual phone number before you run):

!!!warning

This phone number will be used to identify the user, so it must match the number sent by Twilio. It will be sent in the SessionId field, and it will not start with a “+”. So, for example, if your country code is 1, enter the number as 17407593063, without the leading plus sign.

!!!

aws dynamodb put-item \
    --table-name user-portfolio \
    --item \
      '{"UserId": {"S": "{ YOUR PHONE NUMBER WITH COUNTRY CODE }"}, "AvailableCash": {"N": "1000"}, "StockPortfolio": {"L": []}}'

This scripts creates you an account with no stocks and $1000 available cash.

Since the customers will come to your chatbot via SMS, UserId is used as the unique customer id. In a more complex scenario, you would have a different unique id to identify users. Twilio sends the phone number with the county code so you create your record in the same format for convenience.

Similarly, to create the stock prices table, run the following command:

aws dynamodb create-table \
     --table-name stock-prices \
     --attribute-definitions \
         AttributeName=Name,AttributeType=S \
     --key-schema \
         AttributeName=Name,KeyType=HASH \
     --provisioned-throughput \
         ReadCapacityUnits=5,WriteCapacityUnits=5 \
     --table-class STANDARD

Then, run the following to add some stock prices:

aws dynamodb put-item --table-name stock-prices --item '{"Name": {"S": "Apple"}, "Price": {"N": "144.00"} }'
aws dynamodb put-item --table-name stock-prices --item '{"Name": {"S": "Alphabet"}, "Price": {"N": "96.00"} }'
aws dynamodb put-item --table-name stock-prices --item '{"Name": {"S": "Microsoft"}, "Price": {"N": "144.00"} }'
aws dynamodb put-item --table-name stock-prices --item '{"Name": {"S": "Tesla"}, "Price": {"N": "182.00"} }'
aws dynamodb put-item --table-name stock-prices --item '{"Name": {"S": "Twilio"}, "Price": {"N": "46.00"} }'

Now that the database is ready, prepare the IAM roles and policies that your Lambda function will need. The easiest way to set those up is to run the following commands when you’re in the root of the cloned project:

cd src/StockBrokerBot/StockBrokerBot.ChatbotLambda/Setup/
aws iam create-role --role-name stockbrokerbot-lambda-role --assume-role-policy-document file://LambdaBasicRole.json
aws iam attach-role-policy --role-name stockbrokerbot-lambda-role --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam put-role-policy --role-name stockbrokerbot-lambda-role --policy-name dynamodb-table-access --policy-document file://LambdaDynamoDBAccessPolicy.json

LambdaBasicRole.json contains the role for the Lambda function by assuming Lambda service role. Then you attach AWS-managed AWSLambdaBasicExecutionRole policy that grants access to CloudWatch logs. Then, you attach the custom policy specified in LambdaDynamoDBAccessPolicy.json file that grants permissions to access the two DynamoDB tables you created earlier.

Enough with the infrastructure stuff; now it’s time to write some code!

Your Lambda function will receive events from the Amazon Lex service and will use the Amazon DynamoDB service to read/write data. In your terminal, navigate to the root of the Lambda project (src/StockBrokerBot/StockBrokerBot.ChatbotLambda) and run the following commands to add the necessary NuGet packages to your project:

dotnet add package Amazon.Lambda.LexV2Events
dotnet add package AWSSDK.DynamoDBv2

In the Lambda project, create a new directory called IntentProcessors, and under it, a file called AbstractIntentProcessor.cs and set its contents as shown below:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
namespace StockBrokerBot.ChatbotLambda.IntentProcessors;
public abstract class AbstractIntentProcessor
{
    internal const string MessageContentType = "PlainText";
    internal const string IntentStateFulfilled = "Fulfilled";
    internal const string IntentStateFailed = "Failed";
    internal const string DialogActionClose = "Close";
    public abstract Task<LexV2Response> Process(LexV2Event lexEvent, ILambdaContext context);
    protected LexV2Response Close(string intentName, Dictionary<string, string> sessionAttributes, string fulfillmentState, string responseMessage)
    {
        return new LexV2Response
        {
            SessionState = new LexV2SessionState
            {
                Intent = new LexV2Intent { Name = intentName, State = fulfillmentState },
                SessionAttributes = sessionAttributes,
                DialogAction = new LexV2DialogAction  { Type = DialogActionClose }
            },
            Messages = new List<LexV2Message>
            {
                new()
                {
                    ContentType = MessageContentType,
                    Content = responseMessage
                }
            }
        };
    }
}

The abstract class leaves the Process method abstract to be implemented by the inheriting intent processors. It also contains the constants and Close method, which is shared among all the intent processors, so they are placed in the base class to avoid repetition.

In the demo application, you used two JSON-based data providers to manage user portfolio and stock price data. This approach doesn’t work with Lambda functions, as the JSON files will be gone when the function returns. Every time a new copy will be created from scratch, which doesn’t work for databases.

To persist data, you will need DynamoDB providers. Similar to the demo console application, create a directory named Persistence and, under it, create a new file called PortfolioDynamoDBDataProvider.cs.

Update the code as shown below:

using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using StockBrokerBot.Core.Entities;
using StockBrokerBot.Core.Persistence;
namespace StockBrokerBot.ChatbotLambda.Persistence;
public class PortfolioDynamoDBDataProvider : IPortfolioDataProvider
{
    private AmazonDynamoDBClient _dynamoDbClient;
    private DynamoDBContext _dynamoDbContext;
    public PortfolioDynamoDBDataProvider()
    {
        _dynamoDbClient = new AmazonDynamoDBClient();
        _dynamoDbContext = new DynamoDBContext(_dynamoDbClient);
    }
    public async Task<UserPortfolio> GetUserPortfolio(string userId)
    {
        return await _dynamoDbContext.LoadAsync<UserPortfolio>(userId);
    }
    public async Task SaveUserPortfolio(UserPortfolio userPortfolio)
    {
        await _dynamoDbContext.SaveAsync(userPortfolio);
    }
}

Create another file called StockMarketDynamoDBDataProvider.cs and update the code:

using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using StockBrokerBot.Core.Entities;
using StockBrokerBot.Core.Persistence;
namespace StockBrokerBot.ChatbotLambda.Persistence;
public class StockMarketDynamoDBDataProvider : IStockMarketDataProvider
{
    private AmazonDynamoDBClient _dynamoDbClient;
    private DynamoDBContext _dynamoDbContext;
    public StockMarketDynamoDBDataProvider()
    {
        _dynamoDbClient = new AmazonDynamoDBClient();
        _dynamoDbContext = new DynamoDBContext(_dynamoDbClient);
    }
    public async Task<decimal> GetStockPrice(string name)
    {
        var stock = await _dynamoDbContext.LoadAsync<Stock>(name);
        return stock.Price;
    }
}

Now you can implement your first concrete intent processor. Create a new file under the IntentProcessors directory called CheckStockPriceIntentProcessor.cs with the following code:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
using StockBrokerBot.ChatbotLambda.Persistence;
using StockBrokerBot.Core.Services;
namespace StockBrokerBot.ChatbotLambda.IntentProcessors;
public class CheckStockPriceIntentProcessor : AbstractIntentProcessor
{
    public override async Task<LexV2Response> Process(LexV2Event lexEvent, ILambdaContext context)
    {
        var slots = lexEvent.SessionState.Intent.Slots;
        var requestedStockName = slots["stockName"].Value.InterpretedValue;
        var stockMarketService = new FluctuatingStockMarketService(new StockMarketDynamoDBDataProvider());
        var price = await stockMarketService.GetStockPrice(requestedStockName);
        var responseMessage = $"Current price of {requestedStockName} is ${price:N2}";
        return Close(
            lexEvent.SessionState.Intent.Name,
            lexEvent.SessionState.SessionAttributes,
            IntentStateFulfilled,
            responseMessage
        );
    }
}

Lex sends the stockName in the slots dictionary. After getting that value, you pass it on to the StockMarketService, format the output, and send it back to Lex to deliver to the user.

Next, implement the get user portfolio intent. Create a file named GetPortfolioIntentProcessor.cs under the IntentProcessors directory and update the code to:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
using StockBrokerBot.ChatbotLambda.Persistence;
using StockBrokerBot.Core.Services;
namespace StockBrokerBot.ChatbotLambda.IntentProcessors;
public class GetPortfolioIntentProcessor : AbstractIntentProcessor
{
    public override async Task<LexV2Response> Process(LexV2Event lexEvent, ILambdaContext context)
    {
        var userId = lexEvent.SessionId;
        var userPortfolioService = new PortfolioService(
                new FluctuatingStockMarketService(new StockMarketDynamoDBDataProvider()), 
                new PortfolioDynamoDBDataProvider()
        );
        var userPortfolio = await userPortfolioService.GetUserPortfolio(userId);
        return Close(
            lexEvent.SessionState.Intent.Name,
            lexEvent.SessionState.SessionAttributes,
            IntentStateFulfilled,
            userPortfolio.ToString()
        );
    }
}

Twilio sends the user’s phone number to Lex and it sends it to your Lambda function in the SessionId field. You use it to fetch the user portfolio and send it back to the user.

!!!info

Constructing complex objects manually is not ideal. Setting up Dependency Injection is left out as it’s not the focus of this project. You can take a look at this article and implement DI as an improvement.

!!!

Next, create a new intent processor under the IntentProcessors directory called BuyStocksIntentProcessor.cs with the following code:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
using StockBrokerBot.ChatbotLambda.Persistence;
using StockBrokerBot.Core.Services;
namespace StockBrokerBot.ChatbotLambda.IntentProcessors;
public class BuyStocksIntentProcessor : AbstractIntentProcessor
{
    public override async Task<LexV2Response> Process(LexV2Event lexEvent, ILambdaContext context)
    {
        var slots = lexEvent.SessionState.Intent.Slots;
        var requestedStockName = slots["stockName"].Value.InterpretedValue;
        var numberOfShares =  decimal.Parse(slots["numberOfShares"].Value.InterpretedValue);
        var userId = lexEvent.SessionId;
        var userPortfolioService = new PortfolioService(new FluctuatingStockMarketService(new StockMarketDynamoDBDataProvider()), new PortfolioDynamoDBDataProvider());
        try
        {
            var updatedPortfolio = await userPortfolioService.BuyStocks(userId, requestedStockName, numberOfShares);
            var responseMessage = $"Your request has been fulfilled. {updatedPortfolio}";
            return Close(
                lexEvent.SessionState.Intent.Name,
                lexEvent.SessionState.SessionAttributes,
                IntentStateFulfilled,
                responseMessage
            );
        }
        catch (Exception e)
        {
            var responseMessage = $"Error while buying stock: {requestedStockName}. {e.Message}. Call us at +0800 555-555 if the problem persists.";
            return Close(
                lexEvent.SessionState.Intent.Name,
                lexEvent.SessionState.SessionAttributes,
                IntentStateFailed,
                responseMessage
            );
        }
    }
}

And implement the final intent for selling stocks by creating a new file called SellStocksIntentProcessor.cs under the IntentProcessors directory.

Update the code as shown below:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
using StockBrokerBot.ChatbotLambda.Persistence;
using StockBrokerBot.Core.Services;
namespace StockBrokerBot.ChatbotLambda.IntentProcessors;
public class SellStocksIntentProcessor : AbstractIntentProcessor
{
    public override async Task<LexV2Response> Process(LexV2Event lexEvent, ILambdaContext context)
    {
        var slots = lexEvent.SessionState.Intent.Slots;
        var requestedStockName = slots["stockName"].Value.InterpretedValue;
        var numberOfShares =  decimal.Parse(slots["numberOfShares"].Value.InterpretedValue);
        var userId = lexEvent.SessionId;
        var userPortfolioService = new PortfolioService(new FluctuatingStockMarketService(new StockMarketDynamoDBDataProvider()), new PortfolioDynamoDBDataProvider());
        try
        {
            var updatedPortfolio = await userPortfolioService.SellStocks(userId, requestedStockName, numberOfShares);
            var responseMessage = $"Your request has been fulfilled. {updatedPortfolio}";
            return Close(
                lexEvent.SessionState.Intent.Name,
                lexEvent.SessionState.SessionAttributes,
                IntentStateFulfilled,
                responseMessage
            );
        }
        catch (Exception e)
        {
            var responseMessage = $"Error while selling stock: {requestedStockName}. {e.Message}. Call us at +0800 555-555 if the problem persists.";
            return Close(
                lexEvent.SessionState.Intent.Name,
                lexEvent.SessionState.SessionAttributes,
                IntentStateFailed,
                responseMessage
            );
        }
    }
}

Most of the business logic is defined in the core library so what these intents do is to collect data from the user (via Lex and Twilio) and call the corresponding method of the services.

Finally, update your Function.cs as shown below to tie them all together:

using Amazon.Lambda.Core;
using Amazon.Lambda.LexV2Events;
using StockBrokerBot.ChatbotLambda.IntentProcessors;
// Assembly attribute to enable the Lambda function's JSON input to be converted into a .NET class.
[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]
namespace StockBrokerBot.ChatbotLambda;
public class Function
{
    public async Task<LexV2Response> FunctionHandler(LexV2Event lexEvent, ILambdaContext context)
    {
        AbstractIntentProcessor process = lexEvent.SessionState.Intent.Name switch
        {
            "CheckStockPrice" => new CheckStockPriceIntentProcessor(),
            "GetPortfolio" => new GetPortfolioIntentProcessor(),
            "BuyStocks" => new BuyStocksIntentProcessor(),
            "SellStocks" => new SellStocksIntentProcessor(),
            _ => throw new Exception($"Intent with name {lexEvent.SessionState.Intent.Name} is not supported")
        };
        return await process.Process(lexEvent, context);
    }
}

Now you invoke the correct processor based on the intent name specified in the LexV2Event object.

Now deploy your function to AWS by running the following command in the terminal:

dotnet lambda deploy-function

You have created your bot and backend separately. Now it’s the time to bring them together by pointing your bot to your Lambda function, which you will do in the next section.

Connect your Chatbot to Lambda

Go to the Lex Console. Click Bots, then click StockBrokerBot. Click Aliases link, andon the Aliases page, click the Live alias.

In the languages section, click the English (US) link.

Now you should see a page that allows you to select a Lambda function and version of the function. In the Source list, select StockBrokerBot Lambda function and in the Lambda function version or alias list $LATEST should be automatically selected.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 26

Click the Save button.

Now when a request comes to Live alias, Lex will identify the intent and invoke your Lambda function. It will pass all the slot values and user info in a LexV2Event structure that your Lambda expects.

Almost everything is wired up. What’s left is to allow users to interact with your bot via SMS. Proceed to the next section to integrate with Twilio SMS.

Connect your Chatbot to Twilio

On the left menu, right under Aliases, there is a link to Channel integrations. Click that to list the existing integrations.

Click the Add channel button.

Amazon Lex supports 3 integration platforms: Facebook, Slack and Twilio SMS.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 27

Select Twilio SMS.

In the Integration configuration section, enter TwilioIntegration as the name, select Live in the Alias list and English (US) in the language list.

In the Additional configuration section, you will need your Twilio Account SID and Authentication token.

Open the Twilio Console. On the main page, you should see the Account Info section.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 28

Copy your Account SID and Auth Token values and paste them in the corresponding inputs in the AWS console.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 29

Click the Create button.

The Twilio SMS integration should now appear in the list.

Click the channel name to view the details.

Scroll down to the Callback URL section.

You should see an auto-generated webhook URL that Lex expects Twilio to post data to.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 30

To complete the integration, copy the link and go to the Twilio console. Select your account, and then click Phone Numbers → Manage → Active Numbers on the left pane. (If Phone Numbers isn’t on the left pane, click Explore Products and then on Phone Numbers.)

Click the phone number you want to use for your project and scroll down to the Messaging section.

In the “A MESSAGE COMES IN” section, select Webhook and paste the callback URL into the input field. Select HTTP POST in the next dropdown.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 31

Click the Save button at the bottom of the screen.

Test your chatbot via SMS

Finally, it’s time to test your chatbot.

From your phone, send an SMS to your Twilio Phone Number with the following message: check price. You should get a response asking for the stock name. Send the name of one of the stocks in your database such as Tesla. It also understands the stock name directly as it’s in your utterance list. So you can simplify it by sending the stock name directly and you should still get an answer.

Create an SMS chatbot using C#, Amazon Lex, and Twilio SMS - image 32

Now send the following message “buy shares” and your bot should reply by asking the stock name first and then the number of shares. You should see your updated portfolio after the stocks have been bought for you.

You can play around with different utterances and intents.

Conclusion

In this tutorial, you learned how to implement your chatbot from scratch using C#. You also used the Twilio SMS integration with your Amazon Lex chatbot to allow users to interact with your bot via SMS.

A chatbot can be very useful to automate some processes saving time and money for your business. It’s also beneficial for the users as they can use your system outside of business hours.

If you’d like to keep learning, I recommend taking a look at these articles: