0 Comments Posted in:

Azure Blob Storage provides the concept of “shared access signatures”, which are a great way to grant time-limited access to read from (or write to) a specific blob in your container.

“SAS” vs “SAS Token” vs “SAS URI”?

The terminology is confusing, as “SAS” on its own can be used to refer to the entire “SAS URI” or sometimes the “SAS Token”, or even just the “signature”. Here's my understanding of what the terms are:

Here is an example of a SAS URI. This is a full URI that can be used to access a blob:

https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D

Here I have highlighted just the SAS Token portion of the SAS URI. This is the query string appended to the blob’s URI:

https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D

Here I have highlighted just the signature only. This is calculated from the rest of the SAS URI, and requires the Storage Account connection string to calculate. Note that creating this signature is purely in-memory operation – as long as you have the Storage Account connection string, you can generate one without the target blob needing to exist, or needing access to the Storage Account REST API.

https://myaccount.blob.core.windows.net/sascontainer/sasblob.txt?sv=2015-04-05&st=2015-04-29T22%3A18%3A26Z&se=2015-04-30T02%3A23%3A26Z&sr=b&sp=rw&sip=168.1.5.60-168.1.5.70&spr=https&sig=Z%2FRHIX5Xcg0Mq2rqI3OlWTjEg2tYkboXr1P9ZUXDtkk%3D

SAS Usage Guidelines

Shared access signatures can be a great way to share files between microservices in a larger application, as well as making files available to end users for download or display in a webpage. But they can cause problems if used incorrectly. So here are a few guidelines I shared with my development team recently, that might be relevant for your projects too.

  1. Prefer to pass around full SAS URIs. Sometimes I see code that passes just the SAS token, and the name of the file, but this means that the consuming code has to make several assumptions to regenerate the full SAS URI, including the name of the Storage Account. This can also cause problems if you want to use the Azure Storage Emulator whose URI structure is different. By providing a full SAS URI, the receiving code can be completely agnostic about where the file is hosted. For example, if it points at an Amazon S3 Bucket instead, your consuming code won't need to change at all.

  2. Keep the lifetime short, (but not too short). Obviously, from a security perspective, you want to keep SAS lifetimes as short as possible. It does need to be at least long enough to allow the recipient to download the item. But if you put a SAS URI in a queue message, then the lifetime of the SAS should not be less than the TTL (time to live) of the queue message. Otherwise, if the message sits in a queue for a couple of days before being read, it will be useless by the time it is consumed.

  3. Avoid long-lived SAS tokens. It can be tempting to create very long-lived SAS tokens if you want to share an item long-term, but this is generally a bad idea. First, the obvious issue is that there is a greater window of time when it could be accessed if it falls into the wrong hands. But second, SAS tokens become invalidated whenever you cycle your Storage Account keys, so the recipient of the SAS token will need a way to refresh it anyway if they really do need long-term access to the file. Which brings us to...

  4. Generate SAS tokens on-demand wherever possible. Generally, rather than creating a long-lived SAS token and storing it in a database, it's better to have an on-demand process for generating a SAS token at the point the blob access is needed. This allows you to keep the durations short.

  5. Don't bother retrying access denied errors. If you're writing code that attempts to use a SAS URI, and you get access denied, consider it a fatal error - it more than likely means it has expired. (BTW, when I create a SAS token I usually set the start time to five minutes ago, to minimize the chance that system clock inaccuracy results in a SAS token that isn't valid yet)

  6. Avoid issuing container-level SAS tokens. Obviously this one depends on what else might be in the container, but generally it's better to use the principle of "least privilege", and generate a SAS token only for the specific blobs that are required. I'd like to see a SAS feature in the future where you could grant access with a wildcard to all blobs matching a prefix, but I don't think that is currently possible. (If you do need to create a container level SAS token, you I created a tutorial here)

  7. Never write SAS tokens into log messages. Log messages are seen by support staff, who should not have access to private customer data. I like to strip off the SAS token portion of the URI before writing it to the logs, allowing support to see which file had a problem, but not the contents of the file.


0 Comments Posted in:

Note: this is an updated version of my earlier post Managing Azure Functions Keys to use the new Functions ARM APIs

Azure Functions allows you to protect access to your HTTP triggered functions by means of authorization keys. For each function you can choose an "authorization level":

  • anonymous means no API key is required,
  • function means an API key is required.
    • This can be a function-specific key (each function can have its own collection of keys)
    • Or it can be one of the "host" keys which are keys that can be used for all functions with this authorization level
  • admin means you are required to provide the special "master" host key, which is a single key that can be used to call any function in your function app.

To call a protected function you either provide the key as a query string parameter (in the form code=<API_KEY>) or you can provide it as a HTTP x-functions-key header.

Accessing and managing keys in the portal

The Azure portal makes it nice and simple to discover the values these keys. First of all, if we navigate to any HTTP-triggered function in the portal, you'll see a "Get Function URL" link:

image

When we click it, it constructs the URL we need to call including the code query string parameter. This dialog also lets us access values for both types of key - the "function" keys specific to this function, and the "host" keys that can be used on all functions, including the special "_master" host key. You can read more about these key types here.

image

We can manage the keys for an individual function by heading into the "manage" tab for that function:

image

In here we get the ability to view, renew or revoke each individual function key as well as the host keys. You can create multiple function or host keys, which is great as it allows you to provide separate keys to every client you want to grant access to your function, or to implement key cycling.

image

Using the Functions ARM APIs to manage keys

Although its very convenient to manage keys in the portal, before long you'll probably want to manage these values programatically. One way to do that is with the key management API which is what I showed how to use in the original version of this tutorial.

However, it was quite tricky to use, and recently some new Functions ARM APIs were added that be used to do all the same things, and are easier to use.

Let's see how we can get the values of function keys, as well as generate our own, and update them with new values.

Listing keys for a specific function

To use these APIs you need to know how to generate an ARM resource id for your Function App. This requires the Function App name, the Resource Group name and the Azure subscription id.

Here's how to build the resource id for a Function App, using the Azure CLI's az account show method to look up the subscription id from the subscription name.

$subscriptionName = "My Azure Subscription"
$subscriptionId = az account show -s "$subscriptionName" --query id -o tsv
$resourceGroup = "MyResourceGroup"
$webAppName = "myfunctionapp"
$resourceId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/Microsoft.Web/sites/$webAppName"

Next, we're going to call the listKeys endpoint to list the keys for a function. To start with, I'll show how we can do this with PowerShell's Invoke-RestMethod.

First, we need to get hold of a valid access token to call the ARM APIs, which we can get with Azure CLI's very convenient az account get-access-token. Then we can use that to POST to the listKeys endpoint for the specific function we want.

# get an access token
$accessToken = az account get-access-token --query accessToken -o tsv
$functionName = "MyFunctionName"
$listFunctionKeysUrl = "https://management.azure.com$resourceId/functions/$functionName/listKeys?api-version=2018-02-01"
$functionKeys = Invoke-RestMethod -Method Post -Uri $listFunctionKeysUrl `
    -Headers @{ Authorization="Bearer $accessToken"; "Content-Type"="application/json" }

This will return us a JSON object containing all the keys defined for your function. Typically there will just be one called default, so you can access the value with $functionKeys.default.

There is, however, an easier way for users of the Azure CLI. You can use the new preview az rest command which will get hold of the access token for you. So we can get the keys for a function with a single line:

az rest --method post --uri "https://management.azure.com$resourceId/functions/$functionName/listKeys?api-version=2018-02-01"

Listing host keys

You can also ask for the "host keys", with a similar technique:

az rest --method post --uri "$resourceId/host/default/listKeys?api-version=2018-11-01"

This will return something looking like this (no these aren't my actual keys!):

{
  "functionKeys": {
    "default": "VyEOe9oIHkIz6sj+Are3ffHcP7ptHKAidUAxgDMkdSSaYuGDraek5Q=="
  },
  "masterKey": "N9scClcIIRhRYGJN2JHfJatPbkmjvb08QZNixfwuoJrRIrj12iryKA==",
  "systemKeys": {
    "durabletask_extension": "vlNYs6GwWntJ7GXb9/MJ4A36XzM4a+06GFWZrPUqpQL5EBhYA0cHUg==",
    "eventgrid_extension": "UYe7XF4rpC0yhokdOIWEsMWfWQFlpLtf4usgZJq1tC6RHzr0BUKCrw=="
  }
}

Let me briefly explain what these all are:

  • functionKeys are the "host" keys shown in the portal. They can be used to call any HTTP triggered function. There will be a default key out of the box, and you can add your own.
  • masterKey is a special key (shown as _master in the portal) that can be used to access admin authorization level functions. This key should be guarded very carefully.
  • systemKeys contains keys used by any extensions you have installed. In this Function App I have the Durable Functions and Event Grid extensions installed, and they both add their own keys. It's really useful to access these if you want to call the Durable Functions REST API, or if you want to create a new Event Grid subscription.

By the way, if you're wondering how I knew what values are valid for the api-version (which is required on all REST API calls), I found this helpful article from Tobias Zimmergren, showing how to list the available versions with Azure PowerShell. I was using the newer Az PowerShell module, so I got the list of API versions like this:

((Get-AzResourceProvider -ProviderNamespace Microsoft.Web).ResourceTypes `
    | Where-Object ResourceTypeName -eq sites).ApiVersions

Adding and removing a function key

To add a new function key, we perform a HTTP PUT to the key we want to add (or update). You pass in an object containing the key name and value to be updated.

$keyName = "MyFunctionKey"
$payload = (@{ [email protected]{ name=$keyName; value="abcd1234" } } `
    | ConvertTo-Json -Compress).Replace('"', '\"')
az rest --method put `
    --uri "$resourceId/functions/$functionName/keys/$($keyName)?api-version=2018-11-01" `
    --body "$payload"

To remove the key, it's just a HTTP DELETE method to the same endpoint:

az rest --method delete `
    --uri "$resourceId/functions/$functionName/keys/$($keyName)?api-version=2018-11-01"

Adding and removing a host key

The technique for replacing host keys is the same, just now we call the /host/default/functionkeys endpoint. Here I'm adding a new host key.

$keyName = "MyHostKey"
$payload = (@{ [email protected]{ name=$keyName; value="efgh5678" } } | ConvertTo-Json -Compress).Replace('"', '\"')
az rest --method put --uri "$resourceId/host/default/functionkeys/$($keyName)?api-version=2018-11-01" --body "$payload"

I assume you can also use this technique to update the "system keys" and master key although I've not tried that yet. Don't try to delete them though.

To delete the host key we added, it's again very straightforward

az rest --method delete `
    --uri "$resourceId/host/default/functionkeys/$($keyName)?api-version=2018-11-01"

Summary

It's now possible to completely automate the retrieval, creation and update of keys using the new Functions ARM APIs. This is a lot simpler than the previous key management API, and has the benefit of being usable from ARM templates as well. Check out this GitHub thread for some examples of how to use these APIs in your ARM templates.

Want to learn more about how easy it is to get up and running with Azure Functions? Be sure to check out my Pluralsight courses Azure Functions Fundamentals and Microsoft Azure Developer: Create Serverless Functions

0 Comments Posted in:

It's very common in software development to trade off short term gain against long-term pain. If there's a "quick way" to do get a bug fixed or a feature implemented, then more often than not, developers tend to choose it. This results, over time, in the accumulation of "technical debt", where the decisions made in the past are hindering our progress going forwards.

This is why "technical debt" is often such a controversial topic. Martin Fowler recognizes that not every decision made to introduce technical debt is necessarily a prudent one. Maybe our own naivety and inexperience leads to us "inadvertently" introducing technical debt. Or maybe we just capitulate to the business pressure to get the work done as soon as possible, long-term consequences be damned.

But I would argue, that more often than not, as developers, we believe we are making a justifiable decision. In over 20 years as a software developer I don't recall ever having a discussion saying "let's be lazy and irresponsible and get this finished as quickly as possible". Instead, there is a quite reasonable sounding rationalization every time we choose the quick way.

Here's my list of the top seven justifications for introducing technical debt.

1. "It's urgent!"

Sometimes in software development, it really is urgent. When I worked on embedded software in telecoms hardware, downtime could result in mobile phone networks losing money at an astonishing rate, and their customers potentially unable to communicate urgent messages. So if a critical bug can be fixed with three lines of slightly "hacky" code, then that's probably preferable to spending a week implementing it the "proper way".

2. "We need to be strategic"

Another common reason for rushing through some work quicker than would be ideal is the desire to be "first to market". This is one of the main scenarios Ward Cunningham envisioned when he coined the term "technical debt". Sometimes, the commercial advantage of beating your competitors to market is worth the pain of needing to rewrite large portions of your application further down the road.

3. "It's just an MVP"

Closely related is the fact that often in software development, we're creating a feature that may not end up being used long-term. We think it's what the customers want, but until we get it out there in front of them, we're not sure. So we don't want to waste vast amounts of time perfecting code that may not end up being used by anyone. When we're in this scenario, there's a lot of logic to the idea of creating a "minimum viable product" or a "proof of concept". Of course, we intend to revisit the code in the future and provide a fuller, more robust implementation if the feature proves a success. But it's not uncommon for these bare-bones initial implementations to live a lot longer than we intended.

4. "We need to be pragmatic"

Suppose you're asked to design a new feature and estimate how long it will take to implement. You produce a thorough design considering security, scalability, observability, etc, and estimate that it will take six months development time. This comes as a shock to the business. They wanted the feature, but not that much. You're asked to be "realistic" and "pragmatic" about it. You're reminded that "The perfect is the enemy of the good". And of course, it's true that there's always "more than one way to skin a cat". Maybe there's another approach to the problem that's "good enough" and can be done in half the time.

5. KISS

The "Keep it Simple, Stupid (KISS)" principle is well-known and an important reminder that often in software, the most simple solution is in fact the best one. Often by writing less code, we end up with a better, more maintainable, more performant codebase. Technical debt can be introduced not only by a careless developer overlooking important concerns, but by an over-zealous developer writing copious amounts of bloated and complex code to handle imagined problems that may never occur in real-world systems. Many developers have a tendency to "gold plate" or "over-engineer", but it requires plenty of wisdom and experience to make the right judgment calls to resist this temptation.

6. YAGNI

Another closely related reason for technical debt is the well-meaning tendency to write additional code because we think it will probably be needed in the future. At first glance, this seems to be a wise approach - surely the more we bear the future in mind when we write code, the less technical debt we will introduce. The trouble is, we are typically terrible at predicting the future. For example, I've seen complex localization frameworks added into software that was only ever delivered in English, and fragile configuration DSLs introduced for logic that never changed and would have been far simpler to hard-code. Again, striking the right balance between a strict YAGNI approach of only coding what is absolutely needed now, and strategically designing in accessibility points for features you know are coming soon, is hard to get right.

7. "It's too risky"

This one is perhaps the most common reason for introducing (or more commonly, compounding) technical debt. Say you're working on extending some existing code, and you recognize that it is in dire need of refactoring, or maybe needs an upgrade to use a newer framework version. When this happens you have a great opportunity to pay off some technical debt, improving the overall state of the existing code, before adding your new feature. The trouble is, wide-scale refactoring and upgrading to newer technology stacks are generally disruptive and risky changes. Many developers quite understandably take the safer route of maintaining the status quo, but this results in an even bigger job for whoever does finally attempt to sort out the mess in the future.

Are these valid reasons?

What do all the reasons I gave above have in common? They are all arguably the "right decision" in certain circumstances. But they may all also cause us considerable problems in the future. The challenge with technical debt is that, unlike real-world debt (where your creditors can be relied on to demand repayment), it's not at all obvious which instances "technical debt" will come back to bite you. Sometimes your sub-optimal changes sit harmlessly in source control for decades to come without getting in anyone's way. Whereas other quite seemingly reasonable and pragmatic compromises turn out to be the source of constant pain. It's only with the benefit of hindsight that it becomes clear what decision should have been made.

What can we do about this? Is it hopeless given that you can't know for sure whether these "reasons" are valid until it's too late? A few brief thoughts in conclusion.

  1. Whenever you find yourself citing one of the seven reasons I gave above, try to give at least some thought to how the technical debt you introduce as a result might pose problems in the future, and what you can do to minimise those risks.

  2. Cut the developers who went before you some slack when you're next about to rant about how "stupid" and shortsighted the existing code is. Most likely they were making what seemed at the time to be a sensible and pragmatic choice.

  3. Accept a certain amount of "technical debt" is inevitable on any large, and ensure you incorporate sufficient time to address it and keep it from spiraling out of control. This should be a regular part of all ongoing development, not a one off technical debt "blitz".

Anyway, I know the topic of technical debt always raises some interesting (and sometimes heated) discussion, so I'd love to hear your thoughts on these reasons. This is also a great opportunity for me to say that on October 2nd I'll be speaking about technical debt at Techorama Netherlands. It's a great conference with an excellent lineup again this year, so if you're able to make it

Want to learn more about the problem of technical debt and how you can reduce it? Be sure to check out my Pluralsight course Understanding and Eliminating Technical Debt.