-->

dev sendgrid, dotnet, email, rss, twilio

This article was originally published on the Twilio Blog.

In this article, you will learn how to create a nicely formatted dynamic email using the Twilio Blog RSS feed as source data and send it via the SendGrid API. You will first look into creating the template with test data. Then you will learn how to parse RSS and HTML and send the emails with dynamic data.

Prerequisites

You’ll need the following things for this tutorial:

Create a Dynamic Email Template

Follow the steps below to create the dynamic email template for RSS feed digest email:

Go to the Dynamic Templates dashboard and click Create a Dynamic Template. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 1

Enter the name of your template (e.g. blog-rss-feed-digest-email) and click Create. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 2

Expand your template and click Add Version. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 3

Hover over Blank Template and click the Select button. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 4

Click the Select button in the Design Editor section. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 5

Update Version Name to version-1, and then update the Subject to {{subject}}.

Delete the Unsubscribe module by hovering over it and clicking the trash can icon.

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 6

Confirm the deletion by clicking Confirm button in the dialog box.

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 7

!!!warning

The sample application shown in this article is meant to be used for learning purposes only. It’s meant to send emails to yourself. If you intend to send emails to third parties, make sure to click Learn more button in the dialog or visit SendGrid documentation on Global Unsubscribes and Group Unsubscribes.

!!!

Now click the Build tab and drag the Code module into the design area that says Drag Module Here. How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 8

The edit module screen will automatically appear. Paste the following code inside the editor and click Update.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <style>
        body { background-color: lightyellow; }
        h2, h3 { color: green; margin-left: 40px; }
        table { border-collapse: collapse; }
        tr.separated td { border-top: 1px dashed black; padding: 5px; }
        a { text-decoration: none; }
        .postTitle { font-size: 20px; }
        .readMoreButton {
            background-color: #04AA6D;
            border: none;
            color: white;
            padding: 8px;
            text-align: center;
            text-decoration: none;
            display: inline-block;
            font-size: 12px;
            margin: 4px 2px;
            cursor: pointer;
            border-radius: 2px;
        }
        .headerImage { padding-right: 10px; }
    </style>
</head>
<body>
<div>
    <h2>Hello, {{recipientName}}</h2>
    <h3>Here are the latest blog posts from Twilio Blog:</h3>
    {{#each blogPostList}}
    <table>
        <tr class="separated">
            <td><img class="headerImage" src="{{this.headerImageUrl}}" width="200" height="112"></td>
            <td>
                <a class="postTitle" href="{{this.Link}}"> {{this.title}} </a>
                <p>by <b>{{this.author}}</b> - <b>{{this.publishDate}}</b></p>
                {{#if this.categories}}
                    <p>Categories: {{this.categories}}</p>
                {{/if}}
                <p>{{{this.description}}}</p>
                <p>
                    <a class="readMoreButton" href="{{this.link}}">Read more</a>
                </p>
            </td>
        </tr>
    </table>
    {{/each}}
</div>
<div>
    <h2>Last Build Date: {{lastBuildDate}}</h2>
</div>
</body>
</html>

Click Save in the top menu.

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 9

Handlebars Templating Language

SendGrid uses the Handlebars Templating Language to handle variable substitution and add some logic to the templates. You can find all the supported features on SendGrid documentation: Using Handlebars. You might also want to check out this article which also uses Handlebars templating.

Let me show you some of the Handlebars features used in the dynamic email template.

Conditionals

Twilio blog posts can belong to multiple categories. Also, in some cases, they don’t have any categories. So, instead of showing a blank Categories line in the email, you can hide the line if the post does not have any categories.

To check if a string is empty or not, we can use a simple if statement as shown below:

{{#if this.Categories}}
  <p>Categories: {{this.Categories}}</p>
{{/if}}

If the value of this.Categories variable is an empty string, it evaluates to false. In that case, the p element will be hidden.

HTML Injection

As you will see later in the Parsing HTML section, the full HTML post is included in the XML, and you will parse the first paragraph as HTML. In the email, you need to embed this block as HTML; otherwise, it would look broken if other HTML elements were inside the paragraph.

To inject HTML, you can just use triple curly braces as shown below:

<p>{{{this.Description}}}</p>

!!!warning Keep in mind that whenever you are using three curly braces, the variable will not be encoded and susceptible to HTML injection. If this variable holds user input, this can be risky! Make sure to use the HtmlEncoder in .NET to encode user input before passing it into the three curly braces. !!!

Iterations

In the example, you will send an RSS feed digest, meaning there will be multiple blog post sections in the email. Handlebars supports arrays and iterations. You can access each item in the array by using the each keyword. In the example, you did this for the blogPostList array:

{{#each blogPostList}}
…
<p>{{{this.Description}}}</p>
…
{{/each}}

Between {{#each blogPostList}} and {{/each}}, you can access each item in the array by using the this keyword.

Test the Template

The SendGrid designer allows you to preview the rendered output by using hard-coded test data. This is a handy feature as you get to see all the variable substitutions in action right in the designer.

To test your template, click the Preview button in the top menu. You should see something like this:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 10

Next, click Show Test Data to open the data panel on the left.

I obtained some test data manually from the actual Twilio Blog to make the test more realistic. Paste the following JSON into the data panel:

{
  "subject": "Latest Posts - 07 July 2022",
  "recipientName": "Volkan",
  "lastBuildDate": "7 July 2022, 12:34",
  "blogPostList": [
    {
      "title": "Automatically Forward Text Messages with No Code Using Twilio Studio",
      "link": "https://www.twilio.com/blog/automatically-forward-text-messages-no-code-studio",
      "headerImageUrl": "https://twilio-cms-prod.s3.amazonaws.com/images/Copy_of_C03_Blog_Text_2.width-808.png",
      "description": "This article explains how to forward any incoming text messages sent to your Twilio phone number to another number automatically using a no-code solution called Twilio Studio.",
      "author": "Ashley Boucher",
      "publishDate": "06 July 2022",
      "categories": [
        "Code, Tutorials and Hacks"
      ]
    },
    {
      "title": "Super SIM now offers VPN connectivity for your IoT devices",
      "link": "https://www.twilio.com/blog/vpn-iot-devices",
      "headerImageUrl": "https://twilio-cms-prod.s3.amazonaws.com/images/VPN_-_Social_Banner.width-808.png",
      "description": "I am excited to announce that Super SIM now has VPN (Virtual Private Network) support, enabling you to set up secure private networks between Twilio and your application data centers and have your Super SIM connected devices use these private networks. With regular Internet breakout, the traffic from devices using Super SIM will go over the Internet and get routed to your application data center. When VPN is used, the same traffic is sent over a secure and private tunnel as shown below:",
      "author": "Vijay Devarapalli",
      "publishDate": "06 July 2022",
      "categories": []
    }
  ]
}

You should now see the rendered output on the right-hand side:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 11

As you can see, by using the preview feature, you can see the final output without writing any code and sending any emails. Even though the preview is quite accurate, you might want to see it in your inbox as an email. You can also do that in the designer.

Click the Design button on the top menu.

Then, expand Test Your Email section on the left panel:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 12

By default, the From Address field is populated by the email address you used when you created your SendGrid account. However, you can replace it with your verified sender address if you like.

Fill in the Email Addresses field with your recipient’s email address. You can test up to 10 recipients.

Click Send Test Message button. Then, check your inbox, and you should see an email that looks like this:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 13

It looks like the real thing, except that SendGrid prepends the subject with “Test - “.

So far, you have developed an email template, reviewed the rendered output and sent out actual emails using the designer. Now, it’s time to write some code to send the emails programmatically using the same hard-coded data.

Set up your Project to Send Emails

This tutorial will start from an existing git repository. To get the application up and running, follow the steps below:

Clone the GitHub repository:

git clone https://github.com/Dev-Power/send-rss-feed-digest-email-with-sendgrid-dynamic-templates.git --branch 00-starter-project

Alternatively, you can open the repository, switch to the 00-starter-project branch and then click Code and Download ZIP button.

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 14

As a third option, you can download the zip file by clicking on this link.

Navigate into the project folder:

cd send-rss-feed-digest-email-with-sendgrid-dynamic-templates
cd src/RssFeedDigestEmailer.Cli

To store your SendGrid API key securely, add it to the project user secrets by running the following command:

dotnet user-secrets set SendGridSettings:ApiKey [YOUR_SENDGRID_API_KEY]

Replace [YOUR SENDGRID API KEY] with the SendGrid API key you created earlier.

Update appsettings.json and configure the email settings sections:

"emailSettings": {
    "senderEmailAddress": "",
    "senderDisplayName": "",
    "recipientEmailAddress": "",
    "recipientDisplayName": "",
    "templateId": ""
}

Update

  • senderEmailAddress with your SendGrid sender email address,

  • senderDisplayName with any name that you would like the recipient to see,

  • recipientEmailAddress with the email address you want to email,

  • recipientDisplayName with the name of the recipient,

  • and the templateId with the ID of the template you created earlier. You can obtain the Template ID by expanding it in the dashboard:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 15

Now that the project has been set up, review the important parts of the code.

Currently, the project contains an EmailService class to talk to the SendGrid API using the SendGridClient class. It uses a data provider to get the template data. In this example, you have a JsonDataProvider class that looks like this:

public async Task<object> GetEmailData()
{
    string jsonFilePath = "./Data/DummyData.json";
    string rawContents = await File.ReadAllTextAsync(jsonFilePath);
    return JsonConvert.DeserializeObject(rawContents);
}

It reads the data from a hard-coded JSON file named DummyData.json under the Data folder. The contents of DummyData.json are exactly the same as you used in the designer preview.

Having a separate provider for data makes the email service data-agnostic. The SendTemplatedEmail implementation looks like this:

public async Task SendTemplatedEmail()
{
    var dynamicEmailData = await _dataProvider.GetEmailData();
    var from = new EmailAddress(_emailSettings.SenderEmailAddress, _emailSettings.SenderDisplayName);
    var to = new EmailAddress(_emailSettings.RecipientEmailAddress, _emailSettings.RecipientDisplayName);
    var msg = MailHelper.CreateSingleTemplateEmail(from, to, _emailSettings.TemplateId, dynamicEmailData);
    var response = await _sendGridClient.SendEmailAsync(msg);
    if (response.IsSuccessStatusCode)
    {
        Console.WriteLine("Email has been sent successfully");
    }
}

As the email service does not construct the template data itself, you can use different data by swapping out JsonDataProvider with new classes that implement the IDataProvider interface.

The main program is set up like this:

using IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureHostConfiguration(config =>
    {
        config
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true, false);
    })
    .ConfigureServices((hostBuilderContext, services) =>
    {
        services
            .AddTransient<IEmailService, EmailService>()
            .AddTransient<IDataProvider, JsonDataProvider>();
        services
            .AddSendGrid(options => options.ApiKey = hostBuilderContext.Configuration["SendGridSettings:ApiKey"]);
        services
            .Configure<EmailSettings>(hostBuilderContext.Configuration.GetSection("EmailSettings"));
    })
    .Build();
var emailService = (EmailService) ActivatorUtilities.CreateInstance(host.Services, typeof(EmailService));
await emailService.SendTemplatedEmail();

You can see in the setup, user secrets are added to the configuration by calling the AddUserSecrets method. This is required to read the API key from .NET user secrets as you configured in the previous section.

Also, SendGridClient is added to the Dependency Injection (DI) Container using the SendGrid.Extensions.DependencyInjection NuGet package. This way, you don’t have to instantiate the SendGridClient object manually inside the EmailService.

Finally, note that JsonDataProvider is registered for the IDataProvider interface. When you implement getting dynamic data from the RSS feed, you will only have to change the line below, and you won’t have to touch the EmailService:

.AddTransient<IDataProvider, JsonDataProvider>();

Now that you’ve covered the main parts of the application, go ahead and run by running the command:

dotnet run

If all goes well, you should receive an email shortly that looks like this:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 16

It’s almost identical to the one you sent using the SendGrid dashboard, except that the subject is not prepended with “Test - ”.

Next, you will learn how to get the Twilio Blog RSS feed, obtain relevant data from XML and blog post HTML pages, and prepare the dynamic data for the email template.

Get the Twilio Blog RSS Feed

To follow the code explanations below, check out the latest code. While still in the project folder, run:

git checkout main

Alternatively, follow the instructions below to get to the last version of the code:

Under the Services folder, create a new file named TwilioBlogDataProvider.cs and replace its contents with the following:

using Microsoft.Extensions.Options;
using RssFeedDigestEmailer.Cli.Configuration;
using RssFeedDigestEmailer.Cli.Services.Interfaces;
namespace RssFeedDigestEmailer.Cli.Services;
public class TwilioBlogDataProvider : IDataProvider
{
    private readonly IRssService _rssService;
    private readonly EmailDataSettings _emailDataSettings;
    public TwilioBlogDataProvider(IRssService rssService, IOptions<EmailDataSettings> emailDataSettings)
    {
        _rssService = rssService;
        _emailDataSettings = emailDataSettings.Value;
    }
    public async Task<object> GetEmailData()
    {
        var blogInfo = await _rssService.GetBlogInfo();
        return new
        {
            recipientName = _emailDataSettings.RecipientName,
            subject = $"{_emailDataSettings.SubjectPrefix} - {FormatDate(DateTime.Today)}",
            lastBuildDate = FormatDate(blogInfo.LastBuildDate, showTime: true),
            blogPostList = blogInfo.BlogPosts.Select(b => new
            {
                title = b.Title, 
                link = b.Link, 
                headerImageUrl = b.HeaderImageUrl, 
                description = b.Description,
                author = b.Author,
                publishDate = FormatDate(b.PublishDate),
                categories = string.Join("; ", b.Categories) 
            })
        };
        string FormatDate(DateTime date, bool showTime = false)
        {
            return (showTime) ? date.ToString("dd MMMM yyyy, HH:mm") : date.ToString("dd MMMM yyyy");
        }
    }
}

Under Services/Interfaces, create IRssService.cs and paste the following code:

using RssFeedDigestEmailer.Cli.Models;
namespace RssFeedDigestEmailer.Cli.Services.Interfaces;
public interface IRssService
{
    Task<BlogInfo> GetBlogInfo();
}

Under Services, create RssService.cs and paste the following code:

using System.Xml;
using Microsoft.Extensions.Options;
using RssFeedDigestEmailer.Cli.Configuration;
using RssFeedDigestEmailer.Cli.Models;
using RssFeedDigestEmailer.Cli.Services.Interfaces;
namespace RssFeedDigestEmailer.Cli.Services;
public class RssService : IRssService
{
    private readonly IHtmlService _htmlService;
    private readonly IHttpClientFactory _httpClientFactory;
    private readonly RssSettings _rssSettings;
    public RssService(IHtmlService htmlService, IHttpClientFactory httpClientFactory, IOptions<RssSettings> rssSettings)
    {
        _htmlService = htmlService;
        _rssSettings = rssSettings.Value;
        _httpClientFactory = httpClientFactory;
    }
    public async Task<BlogInfo> GetBlogInfo()
    {
        var httpClient = _httpClientFactory.CreateClient("RssServiceHttpClient");
        using (HttpResponseMessage response = await httpClient.GetAsync(_rssSettings.FeedUrl))
        using (var rawRssFeedStream = await response.Content.ReadAsStreamAsync())
        { 
            var xmlDocument = new XmlDocument();
            xmlDocument.Load(rawRssFeedStream);
            var blogInfo = new BlogInfo();
            XmlNode lastBuildDateNode = xmlDocument.SelectSingleNode("/rss/channel/lastBuildDate");
            blogInfo.LastBuildDate = DateTime.Parse(lastBuildDateNode.InnerText);
            XmlNodeList itemNodeList = xmlDocument.SelectNodes("/rss/channel/item");
            XmlNamespaceManager xmlNamespaceManager = new XmlNamespaceManager(xmlDocument.NameTable);
            xmlNamespaceManager.AddNamespace("dc", "http://purl.org/dc/elements/1.1/");
            for (int i = 0; i < itemNodeList.Count; i++)
            {
                XmlNode titleNode = itemNodeList[i].SelectSingleNode("title");
                XmlNode linkNode = itemNodeList[i].SelectSingleNode("link");
                XmlNodeList categoryNodes = itemNodeList[i].SelectNodes("category");
                XmlNode descriptionNode = itemNodeList[i].SelectSingleNode("description");
                XmlNode authorNode = itemNodeList[i].SelectSingleNode("dc:creator", xmlNamespaceManager);
                var headerImageUrlAndPublishDate = await _htmlService.GetHeaderImageUrlAndPostDate(linkNode.InnerText);
                var blogPost = new BlogPost
                {
                    Title = titleNode.InnerText,
                    Link = linkNode.InnerText,
                    Categories = categoryNodes.Cast<XmlNode>().Select(node => node.InnerText).ToList(),
                    Author = authorNode.InnerText,
                    HeaderImageUrl = headerImageUrlAndPublishDate.Item1,
                    Description = await _htmlService.GetPostIntroduction(descriptionNode.InnerText),
                    PublishDate = DateTime.Parse(headerImageUrlAndPublishDate.Item2)
                };
                blogInfo.BlogPosts.Add(blogPost);
            }
            return blogInfo;   
        }
    }
}

Under Services/Interfaces, create IHtmlService.cs and paste the following code:

namespace RssFeedDigestEmailer.Cli.Services.Interfaces;
public interface IHtmlService
{
    Task<(string, string)> GetHeaderImageUrlAndPostDate(string blogPostUrl);
    Task<string> GetPostIntroduction(string rawPostHtml);
}

Under Services, create HtmlService.cs and paste the following code:

using AngleSharp;
using AngleSharp.Dom;
using AngleSharp.Html.Parser;
using RssFeedDigestEmailer.Cli.Services.Interfaces;
namespace RssFeedDigestEmailer.Cli.Services;
public class HtmlService : IHtmlService
{
    public async Task<(string, string)> GetHeaderImageUrlAndPostDate(string blogPostUrl)
    {
        var config = AngleSharp.Configuration.Default.WithDefaultLoader();
        var address = blogPostUrl;
        var context = BrowsingContext.New(config);
        var document = await context.OpenAsync(address);
        var cellSelector = "#header_image > img";
        var cell = document.QuerySelector(cellSelector);
        var headerImgSrc = cell.Attributes.GetNamedItem("src");
        var publishDateCellSelector = "body > main > section > ul > article > header > div > div.article-authors > span";
        var publishDateCell = document.QuerySelector(publishDateCellSelector);
        return (headerImgSrc?.Value, publishDateCell?.InnerHtml);
    }
    public async Task<string> GetPostIntroduction(string rawPostHtml)
    {
        var parser = new HtmlParser();
        var document = parser.ParseDocument(rawPostHtml);
        var cellSelector = "div:nth-child(1) > p:nth-child(1)";
        var cell = document.QuerySelector(cellSelector);
        return cell?.InnerHtml;
    }
}

Under Configuration, create EmailDataSettings.cs and paste the following code:

namespace RssFeedDigestEmailer.Cli.Configuration;
public class EmailDataSettings
{
    public string RecipientName { get; set; }
    public string SubjectPrefix { get; set; }
}

Under Configuration, create RssSettings.cs and paste the following code:

namespace RssFeedDigestEmailer.Cli.Configuration;
public class RssSettings
{
    public string FeedUrl { get; set; }
}

Update Program.cs as below:

using System.Reflection;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using RssFeedDigestEmailer.Cli.Configuration;
using RssFeedDigestEmailer.Cli.Services;
using RssFeedDigestEmailer.Cli.Services.Interfaces;
using SendGrid.Extensions.DependencyInjection;
using IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureHostConfiguration(config =>
    {
        config
            .AddUserSecrets(Assembly.GetExecutingAssembly(), true, false);
    })
    .ConfigureServices((hostBuilderContext, services) =>
    {
        services
            .AddTransient<IRssService, RssService>()
            .AddTransient<IEmailService, EmailService>()
            .AddTransient<IHtmlService, HtmlService>()
            .AddTransient<IDataProvider, TwilioBlogDataProvider>()
            .AddHttpClient("RssServiceHttpClient");
        services
            .AddSendGrid(options => options.ApiKey = hostBuilderContext.Configuration["SendGridSettings:ApiKey"]);
        services   
            .Configure<EmailSettings>(hostBuilderContext.Configuration.GetSection("EmailSettings"))
            .Configure<RssSettings>(hostBuilderContext.Configuration.GetSection("RssSettings"))
            .Configure<EmailDataSettings>(hostBuilderContext.Configuration.GetSection("EmailDataSettings"));
    })
    .Build();
var emailService = (EmailService) ActivatorUtilities.CreateInstance(host.Services, typeof(EmailService));
await emailService.SendTemplatedEmail();

Update appsettings.json and add the new configuration settings after the emailSettings section:

    "rssSettings": {
        "feedUrl": "https://www.twilio.com/blog/feed"
    },
    "emailDataSettings": {
        "recipientName": "",
        "subjectPrefix": ""
    }

What is RSS?

RSS (Really Simple Syndication) is an easy way to keep up with news and blogs. It’s an XML-based feed generated by the websites. RSS readers periodically download these feeds and compare them to what they have locally. This way, the user can get notifications for the updates.

In this example, you will only look into downloading and parsing an RSS feed.

Downloading RSS Feed

First, you need to find out the address of the RSS feed. These are generally plain XML files hosted on the blog or website. If you know that there is an RSS feed, but you don’t know how to find it, one way to find out is to check the website’s source code. For example, on the Twilio Blog, you can view the source and search for RSS in the code. Next, you should see the link to the feed:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 17

Once you know where to download the feed, the rest is the same as downloading any file from the internet. The project uses the following code snippet to download the RSS feed and create the XmlDocument by loading the XML stream:

var httpClient = _httpClientFactory.CreateClient("RssServiceHttpClient");
using (var response = await httpClient.GetAsync(_rssSettings.FeedUrl))
using (var rssFeedStream = await response.Content.ReadAsStreamAsync())
{ 
    var xmlDocument = new XmlDocument();
    xmlDocument.Load(rssFeedStream);

The _httpClientFactory variable shown above is injected during the program setup:

.AddHttpClient("RssServiceHttpClient");

Parsing XML

After you get the raw XML, the next step is to parse and extract the bits you will use in your dynamic template.

You can find the full RSS 2.0 specification here. In the example project, you will use the main required elements: title, link, and description.

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 18

In the example, the built-in System.XML classes are used to parse the XML. The blog post items are under the /rss/channel/item path, so you get those elements like this:

XmlNodeList itemNodeList = xmlDocument.SelectNodes("/rss/channel/item");

The next step is to loop through the XML nodes in the XmlNodeList object which can be accessed by their index in the array:

for (int i = 0; i < itemNodeList.Count; i++)
{
    XmlNode titleNode = itemNodeList[i].SelectSingleNode("title");
    XmlNode linkNode = itemNodeList[i].SelectSingleNode("link");
    XmlNode descriptionNode = itemNodeList[i].SelectSingleNode("description");
    // ...
}

Other nodes are parsed and used in the example, but for brevity, the code snippet above only shows the required ones.

One thing to note is parsing the elements with namespaces. In the example project, only the author element has a namespace, and that’s why it’s treated a bit differently. For example, an author element looks like this in the RSS feed:

<dc:creator
        xmlns:dc="http://purl.org/dc/elements/1.1/">Firstname Lastname
</dc:creator>

To access this element, first, you need to define an XML namespace:

XmlNamespaceManager xmlNamespaceManager = new XmlNamespaceManager(xmlDocument.NameTable);
xmlNamespaceManager.AddNamespace("dc", "http://purl.org/dc/elements/1.1/");

And in the parsing code, you can access the element like this:

XmlNode authorNode = itemNodeList[i].SelectSingleNode("dc:creator", xmlNamespaceManager);

Parsing HTML

Initially, I was planning to use only the RSS feed to obtain all the data used in the dynamic template. I also wanted to use the blog header image and published date in the digest email. Unfortunately, those bits of information don’t come in the RSS XML data. That’s why I resorted to HTML parsing. The risk of HTML parsing is that Twilio could change its HTML structure and CSS classes at any point, which would break the code.

!!!info

The HTML parsing is implemented by using the AngleSharp library.

!!!

For every blog post in the feed, the sample project parses the post HTML and selects the header image and the publish date:

public async Task<(string, string)> GetHeaderImageUrlAndPostDate(string blogPostUrl)
{
    var config = AngleSharp.Configuration.Default.WithDefaultLoader();
    var address = blogPostUrl;
    var context = BrowsingContext.New(config);
    var document = await context.OpenAsync(address);
    var cellSelector = "#header_image > img";
    var cell = document.QuerySelector(cellSelector);
    var headerImgSrc = cell.Attributes.GetNamedItem("src");
    var publishDateCellSelector = "body > main > section > ul > article > header > div > div.article-authors > span";
    var publishDateCell = document.QuerySelector(publishDateCellSelector);
    return (headerImgSrc?.Value, publishDateCell?.InnerHtml);
}

The important part is finding the CSS selector of the element you’re interested in. This is relatively straightforward with the Developer Tools in your browser.

For example, to get the selector of the header image, open a Twilio blog post. Then right-click on the header image and select Inspect from the context menu. While the element is still selected, right-click again to open the context menu. Click Copy and then Copy selector as shown in the screenshot below:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 19

The selector copied should look like this: #header_image > img

This is the selector used in the project to get a reference to the img element. Then the src attribute is accessed by the following line:

var headerImgSrc = cell.Attributes.GetNamedItem("src");

Similarly, you can obtain the CSS selector for the publish date cell and use it to extract the date as a string.

Another use of HTML parsing is to get the introduction part of the blog post. The entire blog post is published in the XML feed, but it would take too much space to put it all in the email, so I decided to pick the first paragraph element (<p>).

Also, I didn’t want to load the URL for this task as it’s already in the XML data downloaded. So a different approach is used to parse the first paragraph of the blog post, as shown below:

public async Task<string> GetPostIntroduction(string rawPostHtml)
{
    var parser = new HtmlParser();
    var document = parser.ParseDocument(rawPostHtml);
    var cellSelector = "div:nth-child(1) > p:nth-child(1)";
    var cell = document.QuerySelector(cellSelector);
    return cell?.InnerHtml;
}

It’s more concise as it doesn’t need to download the HTML over the network and only uses the raw HTML passed in the rawPostHtml argument.

Putting it All Together: Sending the Latest Posts from the Twilio Blog via Email

Finally, it’s time to reap the rewards of all the preparation work you put in. The example project supports one last command you will look into now: Send email command.

You get all the data to use in the email from the IDataProvider.GetEmailData implementation. In this final version of the sample application, you will use TwilioBlogDataProvider, which in turn uses the RssService by calling the GetBlogInfo method.

var blogInfo = await _rssService.GetBlogInfo();

Next, you prepare the dynamic data to be used in the rendered HTML:

return new
{
    recipientName = _emailDataSettings.RecipientName,
    subject = $"{_emailDataSettings.SubjectPrefix} - {FormatDate(DateTime.Today)}",
    lastBuildDate = FormatDate(blogInfo.LastBuildDate, showTime: true),
    blogPostList = blogInfo.BlogPosts.Select(b => new
    {
        title = b.Title, 
        link = b.Link, 
        headerImageUrl = b.HeaderImageUrl, 
        description = b.Description,
        author = b.Author,
        publishDate = FormatDate(b.PublishDate),
        categories = string.Join("; ", b.Categories) 
    })
};
string FormatDate(DateTime date, bool showTime = false)
{
    return (showTime) ? date.ToString("dd MMMM yyyy, HH:mm") : date.ToString("dd MMMM yyyy");
}

The code that sends the email is quite concise:

var dynamicEmailData = await _dataProvider.GetEmailData();
var from = new EmailAddress(_emailSettings.SenderEmailAddress, _emailSettings.SenderDisplayName);
var to = new EmailAddress(_emailSettings.RecipientEmailAddress, _emailSettings.RecipientDisplayName);
var msg = MailHelper.CreateSingleTemplateEmail(from, to, _emailSettings.TemplateId, dynamicEmailData);
var response = await _sendGridClient.SendEmailAsync(msg);

To send the email, run the application, and the final email looks like this:

How to send RSS feed digest email with C# and SendGrid Dynamic Email Templates - image 20

Now you get the same email but with data downloaded from the Twilio Blog RSS feed and individual blog pages and formatted in one nice digest email.

Conclusion

In this article, you learned how to create a Dynamic Email Template. You created an HTML email with CSS and used Handlebars templating to render the email with test JSON data. You also learned how to retrieve and parse RSS XML feeds as well as basic HTML parsing.

I hope you found this article helpful and interesting. The source code is publicly available, so feel free to download and play at will.

If you enjoyed this article, here are a few articles I’d recommend reading about template-based emails:

dev dotnet, vault, secrets, configuration

This article was originally published on the Twilio Blog.

Configuration management has always been a challenge for developers. It gets especially tricky when it comes to storing sensitive configuration values such as API keys, tokens, certificates, passwords etc. In this article, you will learn how to use Hashicorp Vault with C# .NET to manage your application’s secrets.

Prerequisites

You’ll need the following things in this tutorial:

Problem Statement

Let’s start by implementing a simple application to demonstrate the issue. It’s a simple .NET console application that sends a single SMS via Twilio.

Run the following commands in a terminal to create the project and add the Twilio NuGet package:

mkdir VaultDemo 
cd VaultDemo
dotnet new console
dotnet add package Twilio

Open the project with your IDE and replace the contents of Program.cs with the code below:

using Twilio;
using Twilio.Rest.Api.V2010.Account;
using Twilio.Types;
var accountSid = "{ YOUR TWILIO ACCOUNT SID }";
var authToken = "{ YOUR TWILIO AUTH TOKEN }";
var senderPhoneNumber = "{ SENDER PHONE NUMBER }";
var recipientPhoneNumber = "{ RECIPIENT PHONE NUMBER }";
TwilioClient.Init(accountSid, authToken);
MessageResource.Create(
    body: "Nothing fancy, just a simple SMS.",
    from: new PhoneNumber(senderPhoneNumber),
    to: new PhoneNumber(recipientPhoneNumber)
);

Replace the placeholders with actual values. To obtain your Twilio Account SID, AuthToken, and Twilio Phone Number, log in to the Twilio Console and copy the values shown in the account info section:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 1

For the recipient phone number, use your actual phone number so that you can receive the SMS and confirm the application is working.

Run the application by running the following command in the terminal:

dotnet run

You now have a working, standalone application without any dependencies on external configuration. Hardcoding secrets this way might only work for throw-away code. Even then, it’s ill-advised, and you should never use this approach. If you forget to delete the code, you will expose your secrets in cleartext.

Most applications use a version control system (such as GitHub, GitLab etc). In enterprise, it’s safe to say all code is pushed to a source code repository. If you hardcode your secrets and push your secrets to the version control system, anybody who has access to the code will have access to your secrets too. If you’re working on a public open-source project, you have now exposed your secrets to the entire world.

!!!warning

If you are using Git as your version control system, even if you delete the sensitive data immediately, it will still exist in your git history.

!!!

As a general rule of thumb, putting secrets in your source code is considered to be a terrible practice.

Environment Variables

A better approach is using environment variables. This way, you can completely separate your code from your config. If you subscribe to the Twelve-Factor-App philosophy, you can see they describe this approach as “Store config in the environment”. You can read more about it in their Config section.

Now, update your application and replace the hardcoded values with the following lines:

var accountSid = Environment.GetEnvironmentVariable("TWILIO_ACCOUNT_SID");
var authToken = Environment.GetEnvironmentVariable("TWILIO_AUTH_TOKEN");
var senderPhoneNumber = Environment.GetEnvironmentVariable("SENDER_PHONE_NUMBER");
var recipientPhoneNumber = Environment.GetEnvironmentVariable("RECIPIENT_PHONE_NUMBER");

For macOS and Linux, set the environment variable like this:

export {KEY}={VALUE}

If you’re using PowerShell on Windows or another OS, use this command:

$Env:{KEY} = "{VALUE}"

If you’re using CMD on Windows, use this command:

set "{KEY}={VALUE}"

Replace {KEY} with the name of the configuration/secret key (such as TWILIO_ACCOUNT_SID). Replace {VALUE} with the value of the configuration/secret (the values you hardcoded in the previous example).

Repeat the above for all 4 configuration values (TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_PHONE_NUMBER and RECIPIENT_PHONE_NUMBER).

Run the application and confirm it’s still working. Now you have given yourself the opportunity of checking in your code to the version control system as there are no sensitive values in it anymore.

User Secrets

You can also improve the local development environment security by using .NET user secrets. This way, the secrets are stored in a JSON configuration file in the user profile directory. Since the secrets are persisted, you don’t have to enter them over and over again, whereas with the environment variable you will lose the values if you close the terminal.

To use user secret in the demo application, run the following command:

dotnet user-secrets init

Now you can add the secrets by running the following commands:

dotnet user-secrets set "KEY" "VALUE"

To be able to read these values back, you will need configuration extension NuGet packages from Microsoft. Run the following commands to add those libraries:

dotnet add package Microsoft.Extensions.Configuration
dotnet add package Microsoft.Extensions.Configuration.EnvironmentVariables
dotnet add package Microsoft.Extensions.Configuration.UserSecrets

!!!info

You can also use the appSettings.json file, command-line arguments, and more to read the configuration values. You can read more about various configuration options at configuration in .NET.

!!!

Update Program.cs as shown below:

```csharp hl_lines=”1” using Microsoft.Extensions.Configuration; using Twilio; using Twilio.Rest.Api.V2010.Account; using Twilio.Types; IConfiguration config = new ConfigurationBuilder() .AddUserSecrets(optional: true, reloadOnChange: false) .AddEnvironmentVariables() .Build(); var accountSid = config["TWILIO_ACCOUNT_SID"]; var authToken = config["TWILIO_AUTH_TOKEN"]; var senderPhoneNumber = config["SENDER_PHONE_NUMBER"]; var recipientPhoneNumber = config["RECIPIENT_PHONE_NUMBER"]; TwilioClient.Init(accountSid, authToken); MessageResource.Create( body: "Nothing fancy, just a simple SMS.", from: new PhoneNumber(senderPhoneNumber), to: new PhoneNumber(recipientPhoneNumber) );


Run the application, and it should still work. 

This example leverages both user secrets and environment variables. You can still add environment variables to overwrite the values in user secrets. The last key loaded wins, and all the other ones are overwritten. So be careful when specifying multiple providers. The benefit of this approach is now you are not directly reading from environment variables which means you have more control over where you store your configuration.

Now you’re in a more secure position when storing the secrets locally in your development environment. What about deployments, though? When you deploy your application, it will fail because it won’t find the secrets. You can choose to remotely log in to your servers and set the environment variables or user secrets, but it’s impractical and doesn’t scale.  

A better approach is to use a central, secure place to store your configuration so that all instances of your application can easily read those values.

Many services provide this functionality, such as Azure Key Vault, AWS Secrets Manager, and Hashicorp Vault. In this article, you will learn how to set up HashiCorp Vault using Docker and configure your application to get your configuration from it.

## Set Up Hashicorp Vault

Vault is an open-source project and can be found on Hashicorp’s [GitHub repository](https://github.com/hashicorp/vault). Since it’s open-source, you have a few alternatives for using Vault:

- Clone/fork the GitHub repository and build it yourself

- Download and run one of the installers created for your platform

- Run it in a Docker container

- Sign up and use their cloud-based solution

First 3 options are self-hosted and completely free. For the hosted solution, you can sign up for free and evaluate it for 30 days. In this article, you’ll run it in a Docker container. 

!!!info

The focus of the article is not setting up an enterprise-grade Vault cluster, which would be too complicated to cover in a single article. You will see how to run it for your development environment in a single Docker container, but the principle is the same. So if you later purchase a cloud-hosted solution, all you have to do is change the vault address. You can find out more about Vault pricing and packaging [here](https://www.hashicorp.com/products/vault/pricing).

!!!

Vault is one of the select few official images, and it’s quite popular on Docker Hub:

![How to get secrets from HashiCorp Vault into .NET configuration with C# - image 2](/images/vpblogimg/2026/08/How-to-get-secrets-from-HashiCorp-Vault-into-dotNET-configuration-with-CSharp/02.png)

Run the following command to pull the latest Vault image and run Vault in a container:

```bash
docker run -d -p 8200:8200 --cap-add=IPC_LOCK --name=dev-vault vault

In the command above -d flag indicates to run it in the background. --cap-add=IPC_LOCK prevents sensitive values from being swapped to disk. As the name of the container implies, this is for development only. Everything runs in memory and all the secrets will disappear if you stop the container.

Now, open a browser and go to http://localhost:8200. You should see a sign-in screen like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 3

To get your token, first, find the container id by running the command below:

docker ps | grep vault

You should see something like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 4

The first value you see is the container id (in this example it’s 892327726ea3)

Copy that value and run the command below by replacing { YOUR CONTAINER ID } with the value you copied:

docker logs { YOUR CONTAINER ID }

At the end of the logs you should see your root token:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 5

Copy your root token and use it in the sign-in screen.

!!!info

Vault server starts in a sealed state, meaning it doesn’t know how to decrypt the data. Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault. You can read more about sealing and unsealing here.

!!!

You should see the default secrets engines:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 6

Cubbyhole and key/value secret engines are enabled by default (and they cannot be disabled). The cubbyhole secrets engine is used to store arbitrary secrets. Paths are scoped per token, and no token can access another token’s cubbyhole. In this article, you will use Key/Value secret engine, which is a generic

secret engine.

Click secret to view the existing secrets (which are none at the moment). You should see a screen like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 7

Click the Create secret button.

Set the path to your secret as twilioapp. Add your config values as you did in the previous examples. Your screen should look like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 8

Click the Save button. Now you should see the values saved as Version 1 of your configuration:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 9

Now that your secrets are Vault, it’s time to modify the application to read these values.

Using Vault C# Client

To access Vault with C#, you are going to use a library called VaultSharp. Run the following command to add the NuGet package to your project:

dotnet add package VaultSharp

Set the user secret VAULT_ADDR to http://127.0.0.1:8200.

Set the VAULT_TOKEN user secret to your root token.

!!!warning

In production, it’s more likely your operations team will provide you with a role that has access to the secrets your application needs, so you won’t have to use tokens this way. This is just for demonstration purposes.

!!!

Update Program.cs as shown below:

using Microsoft.Extensions.Configuration;
using Twilio;
using Twilio.Rest.Api.V2010.Account;
using Twilio.Types;
using VaultSharp;
using VaultSharp.V1.AuthMethods;
using VaultSharp.V1.AuthMethods.Token;
using VaultSharp.V1.Commons;
IConfigurationBuilder configBuilder = new ConfigurationBuilder()
    .AddUserSecrets<Program>(optional: true, reloadOnChange: false)
    .AddEnvironmentVariables();
IConfiguration config = configBuilder.Build();
IAuthMethodInfo authMethod = new TokenAuthMethodInfo(config["VAULT_TOKEN"]);
var vaultClientSettings = new VaultClientSettings(config["VAULT_ADDR"], authMethod);
IVaultClient vaultClient = new VaultClient(vaultClientSettings);
Secret<SecretData> kv2Secret = await vaultClient.V1.Secrets.KeyValue.V2
    .ReadSecretAsync(path: "twilioapp", mountPoint: "secret");
configBuilder.AddInMemoryCollection(kv2Secret.Data.Data.ToDictionary(kv => kv.Key, kv => kv.Value.ToString()));
config = configBuilder.Build();
var accountSid = config["TWILIO_ACCOUNT_SID"];
var authToken = config["TWILIO_AUTH_TOKEN"];
var senderPhoneNumber = config["SENDER_PHONE_NUMBER"];
var recipientPhoneNumber = config["RECIPIENT_PHONE_NUMBER"];
TwilioClient.Init(accountSid, authToken);
MessageResource.Create(
    body: "Nothing fancy, just a simple SMS.",
    from: new PhoneNumber(senderPhoneNumber),
    to: new PhoneNumber(recipientPhoneNumber)
);

Run the application again, and you should now be able to get the secrets from your Vault instance.

The implementation above first gets the user secrets to be able to access Vault. Then, reads the secrets from Vault and adds them back to the .NET configuration so that all configuration values can be managed in one place. To make it more reusable, you can refactor it to use an extension method.

!!!info

If you want to implement a configuration provider to retrieve information from Vault, I recommend reading this article from Hashicorp.

!!!

Vault supports various authentication methods such as app role, AWS auth, Azure auth, etc. Since my focus is on the programming side, I used the basic token authentication method. You can find out more about the other auth methods on the library’s GitHub repo.

Now stop the container by running:

docker stop { YOUR CONTAINER ID }

Run the application, and it will get an error as the Vault is not running anymore. Start the same container again by running

docker start { YOUR CONTAINER ID }

Run your application again, and this time it will fail due to the following error: Unhandled exception. VaultSharp.Core.VaultApiException: {“errors”:[“permission denied”]}

If you check the logs of the container again, you will see the root token is different. Copy that one and sign in to your Vault via UI again, and you will see that your previous secrets are gone now. This is, as discussed before, because we didn’t provide a persistence engine, and everything was kept in memory. In the next section, you will learn how to fix this issue.

Persisting Secrets

When you run Hashicorp Vault in dev mode, everything is stored in-memory, and the web UI is enabled automatically. To persist secrets, you need to run Vault in server mode.

Vault has an extendable model and supports various storage providers for persisting secrets. You can use the file system, an RDBMS such as MySQL or MSSQL, a NoSQL database such as CouchDB or Amazon DynamoDB, or even a cloud-based storage service such as Google Cloud Storage or Amazon S3. You can find the full list of supported providers here. All the data will be encrypted at rest (as well as in transit), so even if a 3rd party gains access to the stored secrets, they wouldn’t be able read them. In this article, you will use the local file system to persist your secrets.

If you created a container in the previous section, stop and delete it by running the following commands:

docker stop { YOUR CONTAINER ID }
docker rm { YOUR CONTAINER ID }

Then, create a folder to store the Vault configuration. It can be placed anywhere you like on your filesystem. The following example uses /dev/vault/config under your user’s home directory.

mkdir -p $HOME/dev/vault/config

Create a file named config.hcl under that folder and update its contents as shown below:

ui = true
disable_mlock = true
storage "file" {
  path = "/vault/file"
}
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = "true"
}
api_addr = "http://127.0.0.1:8200"

Then, run the following command to create the Vault instance:

docker run -d -p 8200:8200 --volume $HOME/dev/vault:/vault vault server

Open a browser tab and go to http://127.0.0.1:8200.

!!!warning

Keep in mind that this article is not meant to teach you how to install a production-grade Vault server, as it requires high-security, high-availability, backups, monitoring, etc. This is still meant to be used for local development.

!!!

This time, you will see a more involved setup since you are now running Vault in server mode: How to get secrets from HashiCorp Vault into .NET configuration with C# - image 10

Enter 1 in both key shares and key threshold fields.

You should see a successful initialization screen:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 11

Click the Download keys button and get a copy of your keys. Then click the Continue to Unseal button to proceed.

You will then be prompted your Unseal key:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 12

Open the JSON file you just downloaded which looks like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 13

Copy the value of the keys property (this is an array but since you requested 1 key share, it only has 1 element.

Click the Unseal button.

Now you should be redirected to the sign-in page. Copy your root token from the JSON and use it to log in.

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 14

You must have noticed it looks different from the dev mode. There is no secret engine called secret. To fix this, click Enable new engine.

Select KV and click Next.

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 15

Enter “secret” in the Path field just to match the previous example and click Enable Engine.

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 16

Now click the Create secret button as you did in the previous section and create secrets.

Update your new Vault token by running the following command:

dotnet user-secrets set VAULT_TOKEN { YOUR NEW ROOT KEY }

Replace { YOUR NEW ROOT KEY } with the value you copied from the file you downloaded.

Run your application again and you should receive an SMS as you did before.

Here’s the difference though:

Open a terminal and find the container id by running

docker ps | grep vault

Now run the following command:

docker restart { YOUR CONTAINER ID }

Replace ` { YOUR CONTAINER ID }` with the id you noted from the previous command.

Check the container logs by running

docker logs { YOUR CONTAINER ID }

If you recall, the last time you did this you saw an error. This time you should see something like this:

How to get secrets from HashiCorp Vault into .NET configuration with C# - image 17

So the Vault server is still running.

Go back to the Web UI, and it will ask you for the unseal key again. After you enter the unseal key and the root token, you can see the secrets you entered have persisted successfully.

Conclusion

In this article, I wanted to demonstrate the necessity and benefit of having a centralized, secure system to manage application secrets and configuration. You started from the very primitive hard-coding secrets to code approach to using an external service which is highly reputable and used by many top-level companies all around the world. Granted, your Vault installation is for development only, but the underlying principles still apply to production. I hope you found this article helpful. If you’d like to keep learning about .NET configuration and containerizing applications, I recommend taking a look at these articles:

dev dotnet, sms, web-scraping, twilio

This article was originally published on the Twilio Blog.

As a Raspberry PI fan, I like to read The MagPi Magazine, which is freely available as PDFs. The problem is I tend to forget to download it manually every month, so I decided to automate the process. If Raspberry Pi is not your thing, you should be able to modify the demo application to work for any periodical publication that offers free downloads.

Prerequisites

You’ll need the following things in this tutorial:

Project Overview

First, let’s understand what the demo intends to achieve. The components involved and the workflow looks like this:

Get notified of new magazine issues using web scraping and SMS with C# .NET - image 1

  • The worker service reads a database to get the latest issues it sends notifications for.

  • The worker service fetches the website for the magazine and gets the latest issue number. Then, it compares the latest issue number in the database to the latest issue number on the website. If the numbers are equal, it means there is no new issue. If the latest issue number on the website is greater, then there is a new issue. If there is no new issue, the worker service goes to sleep. If there is a new issue, it gets the cover image and the direct link URLs from the magazine’s website.

  • The worker service calls Twilio API to send an SMS/MMS message.

  • Twilio sends the message to the user.

  • The worker service updates its database with the latest issue to avoid duplicate messages.

Project Implementation

Let’s start by creating the worker service by running the following commands:

mkdir MagazineTracker
cd MagazineTracker
dotnet new worker

Create the Data Layer

First, let’s look into the data layer. The only piece of information that needs to be stored is the latest issue number that the application processed.

Create a folder inside your project named Data. Then, create a file LatestMagazineIssue.cs, that contains a model class for your data. Add the following code:

namespace MagazineTracker.Data;
public class LatestMagazineIssue
{
    public int IssueNumber { get; set; }
}

Then, create a new file IMagazineIssueRepository.cs in the Data folder that holds a repository interface to outline the data operations you’re going to use. Add the following code to the file:

namespace MagazineTracker.Data;
public interface IMagazineIssueRepository
{
    Task<LatestMagazineIssue> GetLatestIssue();
    Task SaveLatestIssue(int latestIssueNumber);
}

The next step is to decide how to store the data. The requirements of this project are very straightforward, so you don’t need a full-fledged database; a simple JSON file will suffice. Go ahead and create a JSON file named db.json under the Data directory. Update its contents as shown below:

{
  "LatestIssueNumber": 0
}

Then, create another file named JsonMagazineIssueRepository.cs in the Data folder which will contain the repository implementation for the JSON file named that implements the previous interface. Update the code as shown below:

using System.Text.Json;
using Microsoft.Extensions.Options;
namespace MagazineTracker.Data;
public class JsonMagazineIssueRepository : IMagazineIssueRepository
{
    private readonly DatabaseSettings _databaseSettings;
    public JsonMagazineIssueRepository(IOptions<DatabaseSettings> databaseSettings)
    {
        _databaseSettings = databaseSettings.Value;
    }
    public async Task<LatestMagazineIssue> GetLatestIssue()
    {
        var dbAsJson = await File.ReadAllTextAsync(_databaseSettings.JsonFilePath);
        var latestIssue = JsonSerializer.Deserialize<LatestMagazineIssue>(dbAsJson);
        return latestIssue;
    }
    public async Task SaveLatestIssue(int latestIssueNumber)
    {
        var dbAsJson = await File.ReadAllTextAsync(_databaseSettings.JsonFilePath);
        var latestIssue = JsonSerializer.Deserialize<LatestMagazineIssue>(dbAsJson);
        latestIssue.IssueNumber = latestIssueNumber;
        dbAsJson = JsonSerializer.Serialize(latestIssue);
        await File.WriteAllTextAsync(_databaseSettings.JsonFilePath, dbAsJson);
    }
}

The JsonMagazineIssueRepository only needs one parameter: The path to the JSON file. You can encapsulate it in a simple class. Create DatabaseSettings.cs under the Data directory with the following code:

namespace MagazineTracker.Data;
public class DatabaseSettings
{
    public string JsonFilePath { get; set; }
}

Then update your appsettings.json file so that it looks like this:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "DatabaseSettings": {
    "JsonFilePath": "./Data/db.json"
  }
}

Finally, for this stage, update Program.cs as shown below:

using MagazineTracker;
using MagazineTracker.Data;
IHost host = Host.CreateDefaultBuilder(args)
    .ConfigureServices((hostBuilderContext, services) =>
    {
        services.AddHostedService<Worker>();
        services.AddTransient<IMagazineIssueRepository, JsonMagazineIssueRepository>();
        services.Configure<DatabaseSettings>(hostBuilderContext.Configuration.GetSection("DatabaseSettings"));
    })
    .Build();
// await host.RunAsync();
var repo = host.Services.GetRequiredService<IMagazineIssueRepository>();
await repo.SaveLatestIssue(120);
var latestIssue = await repo.GetLatestIssue();
Console.WriteLine(latestIssue.IssueNumber);

From line 7 to 9, is where you register your services with the concrete implementations in the DI container. Then the IMagazineIssueRepository service is retrieved to get the latest magazine issue and print it to the console.

Line 12 is commented out temporarily to make the implementation/debugging phase easier. As of now, you don’t need to worry about scheduling. That will come later. So, for now, run the application by

dotnet run

And confirm your output looks like this:

120

Now that you have a working data layer move on to the next section, where you will do some HTML parsing.

HTML Parse the Magazine Page

You need 3 things to get from the magazine website:

  • The latest issue number

  • The URL of the magazine (PDF or other formats)

  • The URL of the cover image (Optional)

Every magazine tracker will work differently but you can combine the requirements above in a single interface so that all the trackers can work in a similar fashion.

Create IMagazineTrackerService.cs for the interface and update its code as shown below:

namespace MagazineTracker;
public interface IMagazineTrackerService
{
    Task<int> GetLatestIssueNumber();
    Task<string> GetLatestIssueCoverUrl();
    Task<string> GetIssuePdfUrl(int issueNumber);
}

All your trackers must implement the IMagazineTrackerService interface.

Now, implement your first tracker by creating a file MagPiTrackerService.cs with the following dummy implementation:

namespace MagazineTracker;
public class MagPiTrackerService : IMagazineTrackerService
{
    public async Task<int> GetLatestIssueNumber()
    {
        throw new NotImplementedException();
    }
    public async Task<string> GetLatestIssueCoverUrl()
    {
        throw new NotImplementedException();
    }
    public async Task<string> GetIssuePdfUrl(int issueNumber)
    {
        throw new NotImplementedException();
    }
}

To do the HTML parsing, you will use a library called AngleSharp. It makes the whole process a lot easier, and it can be added to your project via NuGet by running:

dotnet add package AngleSharp

Now, take a look at where to find the latest issue number. The easiest way to find the latest issue number is by going to the issues page, which looks like this at the time of this writing:

Get notified of new magazine issues using web scraping and SMS with C# .NET - image 2

If you look at the source of the page (Right click and click Show/View Page Source depending on your browser). If you search the phrase “The MagPi issue 121 out now” (replace the number with the one you see on your screen) you should find the relevant area that looks something like this:

```html hl_lines=”3 4 5 6”

The MagPi issue 121 out now!

… ``` This page contains the latest issue number and a URL of the cover image. To parse this page, update the `MagPiTrackerService` code as shown below: ```csharp using AngleSharp; namespace MagazineTracker; public class MagPiTrackerService : IMagazineTrackerService { private const string MagpiRootUrl = "https://magpi.raspberrypi.com"; public async Task GetLatestIssueNumber() { var config = Configuration.Default.WithDefaultLoader(); var context = BrowsingContext.New(config); var document = await context.OpenAsync($"{MagpiRootUrl}/issues/"); var latestCoverLinkSelector = ".c-latest-issue > .c-latest-issue__cover > a"; var latestCoverLink = document.QuerySelector(latestCoverLinkSelector); var rawLink = latestCoverLink.Attributes.GetNamedItem("href").Value; return int.Parse(rawLink.Substring(rawLink.LastIndexOf('/') + 1)); } public async Task GetLatestIssueCoverUrl() { var config = Configuration.Default.WithDefaultLoader(); var context = BrowsingContext.New(config); var document = await context.OpenAsync($"{MagpiRootUrl}/issues/"); var latestCoverImageSelector = ".c-latest-issue > .c-latest-issue__cover > a > img"; var latestCoverImage = document.QuerySelector(latestCoverImageSelector); var latestCoverImageUrl = latestCoverImage.Attributes.GetNamedItem("src").Value; return latestCoverImageUrl; } public async Task GetIssuePdfUrl(int issueNumber) { throw new NotImplementedException(); } } ``` After loading the page with AngleSharp, you have to write your CSS-selector to get the element you’re interested in. In this example, the latest issue number is obtained from the `href` attribute of the `anchor` element (by parsing the number that follows the latest ‘/’ character) Similarly, the cover URL is parsed from the `src` attribute of the `img` element. !!!info Even though both pieces of information are obtained from the same page, they were implemented as separate methods. This might look repetitive, but the reason for this is to accommodate other trackers. Having both the issue number and cover URL on the same page may not be the case for other magazines, so if you combine them into a single method, you might have issues later on with other trackers. !!! To test the latest version, update the Program.cs file as shown below: ```csharp using MagazineTracker; using MagazineTracker.Data; IHost host = Host.CreateDefaultBuilder(args) .ConfigureServices((hostBuilderContext, services) => { services.AddHostedService(); services.AddTransient<IMagazineIssueRepository, JsonMagazineIssueRepository>(); services.AddTransient<IMagazineTrackerService, MagPiTrackerService>(); services.Configure(hostBuilderContext.Configuration.GetSection("DatabaseSettings")); }) .Build(); // await host.RunAsync(); var repo = host.Services.GetRequiredService(); var tracker = host.Services.GetRequiredService(); var latestProcessedIssue = await repo.GetLatestIssue(); var latestIssueNumber = await tracker.GetLatestIssueNumber(); if (latestIssueNumber > latestProcessedIssue.IssueNumber) { Console.WriteLine($"New issue detected: {latestIssueNumber}"); var coverUrl = await tracker.GetLatestIssueCoverUrl(); Console.WriteLine($"Cover URL: {coverUrl}"); } ``` Now the `IMagazineTrackerService` is also configured as a service and retrieved from the service provider. Then `tracker.GetLatestIssueNumber` and `tracker.GetLatestIssueCoverUrl` is used to scrape the data and print it. Run the application, and you should see an output that looks like this: ``` New issue detected: 121 Cover URL: https://magpi.raspberrypi.com/storage/…/MagPi121_COVER_STORE.jpg ``` The third and final piece of information you need is the link to the PDF file. If you click on the “Download Free PDF” link, you get redirected to [https://magpi.raspberrypi.com/issues/121/pdf](https://magpi.raspberrypi.com/issues/121/pdf), which looks like this: ![Get notified of new magazine issues using web scraping and SMS with C# .NET - image 3](/images/vpblogimg/2026/08/Get-notified-of-new-magazine-issues-using-web-scraping-and-SMS-with-CSharp-dotNET/03.png) !!!info I'd strongly recommend everybody to consider donating. This is a great magazine with professional quality, and it's full of valuable knowledge about everything Raspberry Pi. !!! If you click on the "No thanks, take me to the free PDF" link, you get redirected to [https://magpi.raspberrypi.com/issues/121/pdf/download](https://magpi.raspberrypi.com/issues/121/pdf/download), and your download starts automatically. This is done by placing an iframe and setting the src as the link to the URL. If you look at the source code of the download page and search for “iframe”, you should find the relevant code looks like this: ```html
``` To parse this URL, update the `MagPiTrackerService.GetIssuePdfUrl` method as shown below: ```csharp public async Task GetIssuePdfUrl(int issueNumber) { var issueUrl = $"{MagpiRootUrl}/issues/{issueNumber}/pdf/download"; var config = AngleSharp.Configuration.Default.WithDefaultLoader(); var address = issueUrl; var context = BrowsingContext.New(config); var document = await context.OpenAsync(address); var cellSelector = "iframe"; var cell = document.QuerySelector(cellSelector); var iframeSrc = cell.Attributes.GetNamedItem("src").Value; return $"{MagpiRootUrl}/{iframeSrc.TrimStart('/')}"; } ``` Update the test code in Program.cs only to test the latest update: ```csharp … var latestProcessedIssue = await repo.GetLatestIssue(); var latestIssueNumber = await tracker.GetLatestIssueNumber(); if (latestIssueNumber > latestProcessedIssue.IssueNumber) { Console.WriteLine($"New issue detected: {latestIssueNumber}"); var pdfUrl = await tracker.GetIssuePdfUrl(latestIssueNumber); Console.WriteLine($"PDF URL: {pdfUrl}"); } ``` Run the application and confirm you can see the same URL you saw in the download page source: ``` New issue detected: 121 PDF URL: https://magpi.raspberrypi.com/downloads/…/MagPi121.pdf ``` ### Set up Twilio to Send SMS Notifications Before implementing the actual notification mechanism, create a new interface to ensure all notification channels work the same. Create a file named INotificationService.cs and update its code like this: ```csharp namespace MagazineTracker; public interface INotificationService { Task SendNewIssueNotification(int issueNumber, string coverUrl, string mediaUrl); } ``` In the demo project, you will implement SMS/MMS notifications using [Twilio Programmable SMS](https://www.twilio.com/docs/sms). Now that you have all the information, you need to deliver this to Twilio so that you can get SMS notifications on your mobile device. To achieve this, first, add Twilio SDK to your project by running: ```bash dotnet add package Twilio ``` You will need your Account SID and Auth Token to be able to talk to the Twilio API. You can find both of these on the welcome page in the account info section when you log in to the [Twilio Console](https://console.twilio.com/): ![Get notified of new magazine issues using web scraping and SMS with C# .NET - image 4](/images/vpblogimg/2026/08/Get-notified-of-new-magazine-issues-using-web-scraping-and-SMS-with-CSharp-dotNET/04.png) To store these values, you can use environment variables or a vault service, but for local development, you can use [dotnet user secrets](https://docs.microsoft.com/en-us/aspnet/core/security/app-secrets). First, you need to initialize user secrets by running ```bash dotnet user-secrets init ``` Then, create two new user secrets called `Twilio:AccountSid` and `Twilio:AuthToken` and set the values: ```bash dotnet user-secrets set Twilio:AccountSid {YOUR TWILIO ACCOUNT SID} dotnet user-secrets set Twilio:AuthToken {YOUR TWILIO AUTH TOKEN} ``` Create a new file called SmsService.cs and add the following code: ```csharp using Microsoft.Extensions.Options; using Twilio.Rest.Api.V2010.Account; using Twilio.Types; namespace MagazineTracker; public class SmsService : INotificationService { private readonly SmsSettings _smsSettings; public SmsService(IOptions smsSettings) { _smsSettings = smsSettings.Value; } public async Task SendNewIssueNotification(int issueNumber, string coverUrl, string mediaUrl) { MessageResource.Create( body: $"Here's the latest issue (#{issueNumber}) of The MagPi Magazine: {mediaUrl}", from: new PhoneNumber(_smsSettings.FromPhoneNumber), to: new PhoneNumber(_smsSettings.ToPhoneNumber), mediaUrl: string.IsNullOrEmpty(coverUrl) ? null : new [] { new Uri(coverUrl) }.ToList() ); } } ``` The SMS message needs to be sent from your Twilio phone number (which you can find right below Account SID and Auth Token on [Twilio Console](https://console.twilio.com/) welcome page). The reason the code checks whether or not `coverUrl` has a value is that some Twilio Phones Numbers don’t support MMS. For example, Twilio Phone Numbers from the United Kingdom (UK) do not support MMS, so my UK number could only send plain SMS. So, if you are not able to send MMS messages, simply send an empty string as the cover URL so that setting the `coverUrl` in your worker service looks like this: ```csharp var coverUrl = String.Empty; ``` Alternatively, you can create a boolean setting such as `includeCoverUrl` to manage this behaviour. To store both from and to phone numbers, update appsettings.json like this: ```json hl_lines="11 12 13 14" { "Logging": { "LogLevel": { "Default": "Information", "Microsoft.Hosting.Lifetime": "Information" } }, "DatabaseSettings": { "JsonFilePath": "./Data/db.json" }, "SmsSettings": { "FromPhoneNumber": "{YOUR TWILIO PHONE NUMBER}", "ToPhoneNumber": "{YOUR ACTUAL PHONE NUMBER}" } } ``` Create a file called SmsSettings.cs with the following class: ```csharp namespace MagazineTracker; public class SmsSettings { public string FromPhoneNumber { get; set; } public string ToPhoneNumber { get; set; } } ``` Finally, update Program.cs to reflect these changes: ```csharp using MagazineTracker; using MagazineTracker.Data; using Twilio; IHost host = Host.CreateDefaultBuilder(args) .ConfigureServices((hostBuilderContext, services) => { services.AddHostedService(); services.AddTransient<IMagazineIssueRepository, JsonMagazineIssueRepository>(); services.AddTransient<IMagazineTrackerService, MagPiTrackerService>(); services.AddTransient<INotificationService, SmsService>(); services.Configure(hostBuilderContext.Configuration.GetSection("DatabaseSettings")); services.Configure(hostBuilderContext.Configuration.GetSection("SmsSettings")); var accountSid = hostBuilderContext.Configuration["Twilio:AccountSid"]; var authToken = hostBuilderContext.Configuration["Twilio:AuthToken"]; TwilioClient.Init(accountSid, authToken); }) .Build(); // await host.RunAsync(); var repo = host.Services.GetRequiredService(); var tracker = host.Services.GetRequiredService(); var notificationService = host.Services.GetRequiredService(); var latestProcessedIssue = await repo.GetLatestIssue(); var latestIssueNumber = await tracker.GetLatestIssueNumber(); if (latestIssueNumber > latestProcessedIssue.IssueNumber) { Console.WriteLine($"New issue detected: {latestIssueNumber}"); var coverUrl = await tracker.GetLatestIssueCoverUrl(); var pdfUrl = await tracker.GetIssuePdfUrl(latestIssueNumber); await notificationService.SendNewIssueNotification(latestIssueNumber, coverUrl, pdfUrl); await repo.SaveLatestIssue(latestIssueNumber); } ``` !!!info Sending a message via WhatsApp works exactly the same way, except you can only use a sandbox environment unless your account is approved. The sandbox session expires after 3 days, so it’s not a great fit for continuous notifications, but if your account is approved already, you can still use `SmsService` without any modifications. All you have to do is replace the “from phone number” with “whatsapp:+xxxxxxxxxxx”, where xxxxxxxxxxx is the number provided to you by Twilio. Also, prefix the “to phone number” with “whatsapp:” !!! Time to test the final version (which also updates the database with the latest issue number). Run the application, and you should receive an SMS/MMS on your phone. My UK Twilio Phone Number doesn’t support MMS. If I try to set the `coverUrl` to the image URL, I get the following exception: ```bash Twilio.Exceptions.ApiException: Number: +44xxxxxxxxxx has not been enabled for MMS ``` So I set the `coverUrl` to empty string as discussed previously and the SMS I receive on my phone looks like this: ![Get notified of new magazine issues using web scraping and SMS with C# .NET - image 5](/images/vpblogimg/2026/08/Get-notified-of-new-magazine-issues-using-web-scraping-and-SMS-with-CSharp-dotNET/05.png) And when I tap on the link, I get this: ![Get notified of new magazine issues using web scraping and SMS with C# .NET - image 6](/images/vpblogimg/2026/08/Get-notified-of-new-magazine-issues-using-web-scraping-and-SMS-with-CSharp-dotNET/06.png) To test the MMS feature, I purchased a US Twilio Phone Number and sent the same message with the actual `coverURL` (meaning reverted the code to its original version: `var coverUrl = await _magazineTrackerService.GetLatestIssueCoverUrl();`). When I send the message from the US phone number, I get this message: ![Get notified of new magazine issues using web scraping and SMS with C# .NET - image 7](/images/vpblogimg/2026/08/Get-notified-of-new-magazine-issues-using-web-scraping-and-SMS-with-CSharp-dotNET/07.png) It shows the text, the full URL to the PDF and a shortened URL of the cover image. In my case, I prefer the original message. Depending on your phone, carrier and the messaging app you use, your experience may vary. I’d recommend playing around with splitting up the notification into multiple messages, such as sending the text in one message and the cover image in another or sending text, cover image, and URL all in different messages. Try it out and decide which format you like the most. ### Schedule the Worker Service You have a working application but it only functions when you run it manually. To automate the process, move the code into the Worker.cs class shown below: ```csharp using MagazineTracker.Data; namespace MagazineTracker; public class Worker : BackgroundService { private readonly ILogger _logger; private readonly IMagazineIssueRepository _magazineIssueRepository; private readonly IMagazineTrackerService _magazineTrackerService; private readonly INotificationService _notificationService; public Worker(ILogger logger, IMagazineIssueRepository magazineIssueRepository, IMagazineTrackerService magazineTrackerService, INotificationService notificationService) { _logger = logger; _magazineIssueRepository = magazineIssueRepository; _magazineTrackerService = magazineTrackerService; _notificationService = notificationService; } protected override async Task ExecuteAsync(CancellationToken stoppingToken) { while (!stoppingToken.IsCancellationRequested) { _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now); var latestProcessedIssue = await _magazineIssueRepository.GetLatestIssue(); var latestIssueNumber = await _magazineTrackerService.GetLatestIssueNumber(); if (latestIssueNumber > latestProcessedIssue.IssueNumber) { _logger.LogInformation("New issue detected: {latestIssueNumber}", latestIssueNumber); var coverUrl = await _magazineTrackerService.GetLatestIssueCoverUrl(); var pdfUrl = await _magazineTrackerService.GetIssuePdfUrl(latestIssueNumber); await _notificationService.SendNewIssueNotification(latestIssueNumber, coverUrl, pdfUrl); await _magazineIssueRepository.SaveLatestIssue(latestIssueNumber); } else { _logger.LogInformation("No new issue is detected."); } await Task.Delay(1000 * 60 * 60, stoppingToken); // Run hourly } } } ``` This way, you can remove all the previous test code and initializations and Program.cs becomes very concise: ```csharp using MagazineTracker; using MagazineTracker.Data; using Twilio; IHost host = Host.CreateDefaultBuilder(args) .ConfigureServices((hostBuilderContext, services) => { services.AddHostedService(); services.AddTransient<IMagazineIssueRepository, JsonMagazineIssueRepository>(); services.AddTransient<IMagazineTrackerService, MagPiTrackerService>(); services.AddTransient<INotificationService, SmsService>(); services.Configure(hostBuilderContext.Configuration.GetSection("DatabaseSettings")); services.Configure(hostBuilderContext.Configuration.GetSection("SmsSettings")); var accountSid = hostBuilderContext.Configuration["Twilio:AccountSid"]; var authToken = hostBuilderContext.Configuration["Twilio:AuthToken"]; TwilioClient.Init(accountSid, authToken); }) .Build(); await host.RunAsync(); ``` Now run the application again (reset the database first to a value lower than the latest issue number), and you should receive an SMS/MMS; your database should be updated with the latest issue number, and your service should wait for 1 hour and then run the code again. You can, of course, change how often you would like to check for new issues by changing the delay. ## Conclusion My favorite projects are the ones that I develop to solve a real problem of mine. This one was a small issue, but I like the idea of automating something that otherwise I’d forget. Even though there is one implementation of a magazine tracker service, you can adapt the existing code for your favorite publication. As long as you add a new class that implements the same interface, you can replace the registration code in Program.cs and your application will start fetching that magazine. The same goes for the notification. You can replace SMS/MMS with email using [SendGrid](https://www.twilio.com/blog/send-emails-using-the-sendgrid-api-with-dotnetnet-6-and-csharp) or [WhatsApp](https://www.twilio.com/blog/send-a-whatsapp-message-with-c-in-30-seconds). If you'd like to keep learning, I recommend taking a look at these articles: - [How to send vCards with WhatsApp using C# and .NET](https://www.twilio.com/blog/send-vcards-with-whatsapp-using-csharp-and-dotnet) - [Send Emails with C#, Handlebars templating, and Dynamic Email Templates](https://www.twilio.com/blog/send-emails-with-csharp-handlebars-templating-and-dynamic-email-templates) - [Render Emails Using Razor Templating](https://www.twilio.com/blog/render-emails-using-razor-templating)