This is part three of a series of articles illustrating how easily you can incorporate the Rewiring America API to your website, research, and more. In the first post, we walk through how to add the Residential Electrification Model (REM) API to your website with HTML and JavaScript. In the second post, we discussed how to incorporate the REM API to your website using React.

In this post, we’ll illustrate some of the ways in which API keys used on websites you build and maintain could be exposed to the world where they could be exploited — and we’ll walk you through some steps you can take to protect them. 

Note that the security approach we will discuss here is not unique to the Rewiring America API. This same approach can and should be used to protect all the API keys you use to access all of the APIs your site calls to render its pages.

API keys and security

When you register for an API key for any service, chances are it comes along with legal terms and conditions that obligate you to take care of the key and not expose it to third parties. As an example, the terms of service for the Rewiring America APIs require you to agree to “handle API keys with care and rotate any key that you believe may have been made public.”

There are several issues with API keys becoming public. For example, if you’re using metered API endpoints — or API endpoints that incur a cost per use — someone with unauthorized access to your key could very easily use up your quota and/or generate a very large bill by repeatedly calling the API with your key. Non-authorized users could also undermine the code, which could affect not only your website but the underlying code as well.

Exposing keys in source code

The various players in the API ecosystem, from API providers and their clients, to source code repositories like GitHub, have come up with a variety of mechanisms to help keep keys safe. That is why, for example, if you attempt to push code to GitHub that contains what looks like an API key from one of the major providers of API infrastructure, you may get a warning like this one:

If you do get such a warning, resist the temptation to check the “I’ll fix it later” button — and try some of the tips below on how to fix it once and for all.

This post is all about how you can avoid ever seeing this pop-up — and ensure your API keys are secure. We will explain how to avoid keeping your keys in your source code, and instead put them in a safe place like the GCP Secret Manager. There are a number of other secret managers out there that you might choose to use, including Google Cloud platform AWS Secrets Manager, which works very similarly to GCP Secret Manager. This post is not meant to compare or review these services. Using any form of API key security puts you miles ahead of keeping it in the source code. The differences between the secret managers are more about usability and ease of integration with other tools you already use in your development, CI/CD, and DevOps. 

Exposing keys in web clients

There are a number of ways that API keys can be exposed to possibly malicious third parties. Even if you would never ignore the warning on GitHub above, or push out code with an exposed key to a public repository, you might not realize your web app or web page-serving infrastructure may be publicly exposing them on the client side.

One of the key architectural patterns on the modern web is that much of the code that renders pages actually runs on the browser. Browsers often download a shell of a page, along with libraries of JavaScript code, and then execute that code to call the API to obtain the data needed to fill out the rest of the page. In other cases, the page may be complete or almost complete from the get-go, but as the user enters information in forms or clicks on buttons, the browser makes calls back to various APIs to get more information to display. This is exactly how the hypothetical Electrification Nation website operates in the demo we walked through in our first post about the REM API. If you haven’t read it yet, we recommend going back and giving it a read. Once you have gone through it and understand the concepts introduced there, you will be ready for everything you will encounter in this post.

As we described in that first post, someone coming to the hypothetical Electrification Nation website would type in their address and home fuel source into a form like this one, in order to find out how much they could save by switching to a heat pump:

When they hit OK, the browser calls the REM API to get estimated savings, and puts the results on the page like this:

All’s well, right? Not necessarily. If you right-click in your browser and select “View Page Source,” you will see that 20 lines into the HTML and JavaScript code that defines the page is this line: “const api_key = “INSERT_YOUR_API_KEY_HERE”

Network API calls

Sometimes, developers try to obscure or encrypt their API keys, but as long as the calls are being generated from a browser, the API keys can be seen in the headers or body of the call as it is being made by the browser. Here is an example of what you can see just by opening Chrome’s built-in developer tools and looking at a request made by the page we just looked at:

We won’t go into the details, but the request failed because it had an invalid key, which is why the URL is red over on the left side under the “Name” heading. On the right side, I scrolled down just enough to see the request headers that were sent along with the API call. The one highlighted in yellow is “Authorization:” and the value is “Bearer INSERT_YOUR_API_KEY_HERE.” If I had changed that on line 20 of the code above to be my actual API key, it would have been exposed here.

If I had obscured the key in some other way but ultimately decoded it and used it to make a call to the REM API, it would be right here, in the same debugging display, for anyone to see.

Server-side middleware

There is a way around this, not only for the REM API key but for every API key you might use on your site. You can write a small piece of code known as middleware. Middleware runs on the server side, not in the browser. That means the mMiddleware has access to the API keys it needs to make things function, but never shares them with the browser where they could be revealed to third parties

There is code running in the browser that still needs the results of API calls that need API keys. So how can it get them? The answer is that it makes an API call of sorts to the middleware, which then looks up the API key and calls the real API and returns the result to the caller in the client.

Before we look at how to implement this, let’s look at what it looks like from the browser side. If we monitor the network tab we will see this:

The URL looks a little strange, but in this case I am running a test server locally, so it is just calling back to the same IP address that is hosting the site. If you look in the header tab you will not find any API key because it isn’t needed to call the middleware. We will see some other values that are temporary keys within the ongoing execution of the app. And if we look at the payload, all we will see is the address and the fuel we selected in the form. But no API keys.

Implementing middleware With Next.js and a secret manager

So now that we have seen what middleware can do in terms of preventing exposure of our API key to the outside world, let’s have a look at how we actually did this. The key component is a file called serverSavings.ts in the app directory. It implements the middleware that queries the REM API. It begins with:

The first line, 'use server'; could almost be missed. But it does something critical. It causes all functions defined in this file to be Next.js server functions. A server function runs entirely on the server, but it can be called asynchronously by code on the client side using normal function call syntax.

Let’s look at one of the functions in this file:

This function takes two arguments, an address and a fuel, and returns expected savings. It gets the expected savings by calling the URL for the REM API. It does this using axios, exactly as our React-based implementation of the Electrification Nation did in our earlier blog post. The difference is that in that implementation, we made this call from the client side, in the browser, where our API key could be exposed.

This function expects to be able to access our API key via a constant called RA_API_KEY which was somehow already set before the function was called. Remember we said that we were going to store our key in the GCP Secret Manager? We won’t go into details on how to do that, because it is well documented, but we will note that we set up a secret called ra-api-key and stored our key in it. On the GCP console page for Secret Manager, it looks like this:

Now we are going to show how we use the SecretManagerServiceClient that we imported to access this key and set the constant RA_API_KEY to its value.

We first set the ID of our project by reading it from the .env environment variable file, which looks like:

You will want to replace 'XXXXXXXXXXXX' with your actual project ID. Next, we create a secret manager client and use it to access the value of the secret. Finally, we set the constant RA_API_KEY to the key by calling the function.

There are some important things to note here. First, the project ID alone is not sufficient to give anyone access to any resources in the project, whether the secret manager or anything else. So although we might not want to post it all over the internet, it’s not especially dangerous if a third-party finds it out. Second, even the project ID never goes to the client side. Finally, this function is not annotated with export so it cannot be called directly from the client side like serverSavings could be.

For development purposes, we can gain access to the secret manager by having authenticated ourselves with gcloud auth login at the command line. For production use, we can deploy the server side in a docker container with appropriate authorization. If you are not sure how to do these things, contact your GCP project administrator.

So that’s pretty much it. We have used a secret manager to secure our API key, and we have structured our code so that it is only visible at runtime on the server side. This protects it from snooping eyes using various debugger features built into the browser to track down the key. But at the same time, our full application still works exactly as we want it to.

Limitations of this approach

We have just demonstrated how to prevent API keys from leaking to the client side. But it is important to recognize that by doing so we have not solved every problem or prevented every possible attack vector that a malicious actor might try against us. The reason we concentrated on protecting our API key is that API keys and similar keys and credentials — for example for database access — often enable access well beyond what one particular web page like the one we used in this demonstration needs. So by preventing the API key from leaking to the client side, we prevent it from being used in any way other than how we use it in the server side code.

What we did not prevent, however, is a malicious person or group overusing it in a DoS attack. Someone could still call our API many, many times in a short period of time just by simulating being a browser running our page. Indeed, if we go back to the developer console where we were looking at the middleware call, we can very easily copy the details of the request in curl or other format simply by right clicking and following the menu.

When we paste this into a browser, we are able to make an exact copy of the request. It looks like:

If we paste this into a shell window and run it (changing the address to an actual valid one) we get legitimate results.

We never had or needed the server-side API key, but we were able to call the middleware directly.

The difference between having the actual API key and this scenario is that if we had the actual API key we could call the various APIs it enables in a wider variety of ways. So while we have eliminated a large class of attacks enabled by possessing the API key, we have not prevented every issue we might face, in particular DoS attacks via the server-side function we set up.

Conclusions

We began with a running example that we originally built to illustrate how to use the Rewiring America REM API and then rebuilt using React. We then showed how neither of these approaches was as secure as it could have been. Prying eyes could still fairly easily find our API key and potentially reuse it maliciously. We then showed how to avoid these problems by never passing it to the client side and instead calling the REM API from the server side. Finally we further secured our key by not leaving it even in server side code, where it could still leak if it were checked into GitHub or any other public repository. Instead, we used the GCP Secret Manager to store it in an encrypted form so that it was only accessible to processes with permission to access our instance of the GCP Secret Manager.

It is important to note that what we did is not unique to API keys for the Rewiring America API, or even unique to the GCP secret manager. There are other secret managers out there with JavaScript clients. And you should use them to secure all of your API keys, not just your Rewiring America API key.

It is also important to note that this does not protect us from every type of attack. It just eliminates a class of attacks that are enabled by having an API key. These tend to be broader and more substantial than what can be accomplished by calling limited server-side middleware functions like the one we created.

All of the code discussed above, along with the rest of the code to run this secure version of the Electrification Nation web site is available in GitHub. If you have questions, comments, or suggestions, or if you find a bug in any of the code above, please start a discussion or open an issue in the GitHub repository where it is hosted. You can also find additional examples of how to use the Rewiring America API in that project.

Check out our other guides on using the Rewiring America API

• Adding the Residential Electrification Model (REM) API to your website

• How to call the Rewiring America Residential Electrification Model (REM) API from React

• How to use the Rewiring America Health Impact API