LetsEncrypt Certificates for Your Firewalls!
Have you wanted to take advantage of free LetsEncrypt certificates for your firewalls, VPN Portals, or network infrastructure? Did the short LetsEncrypt renewal cycle deter you? Perhaps you hit a roadblock automating the process without native LetsEncrypt support on-device? Maybe you are just tired of paying for your certificates?
If you answered “yes” to either of these questions, this article is for you!
In this article, I will demonstrate you how you can automate the LetsEncrypt certificate process for network and security devices. While I will focus on Palo Alto Networks firewalls for the purpose of this demonstration, the following can be adapted to work with almost any network device, and is particularly useful for VPN headend devices (regardless of make and model).
Prerequisites
Before you get started, you will require the following:
- A linux-based container, VM, or host with reachability to your device’s management interface and the following packages installed:
- openssl
- pan-python
- certbot
- (optional, if you are using CloudFlare) certbot-dns-cloudflare
-
sudo apt-get install python-pip certbot openssl
sudo pip install pan-python
sudo pip install certbot-dns-cloudflare
- An account with a certbot-compatible DNS provider.
- A list of certbot DNS plugins can be found here: https://certbot.eff.org/docs/using.html#dns-plugins
- I will use CloudFlare in this example. The dns-cloudflare plugin automates the process of a dns-01 challenge by creating and removing TXT records using the CloudFlare API
- If you use CloudFlare, you will be required to pass in your CloudFlare credentials as a certbot argument. By default, this is a .ini file containing your CloudFlare username and API key.
- To obtain your CloudFlare API key, navigate to your CloudFlare admin panel and select “My Profile” from the upper-right corner.
- Navigate to the “API Tokens” tab. Select “View” next to “Global API Key”. Copy this key into a .cloudflare.ini file.
- It is best practice to ensure this file can only be accessed by your user (or the user cron runs as).
-
dns_cloudflare_email = [email protected] dns_cloudflare_api_key = 0123456789abcdef0123456789abcdef01234567
- An API key for your firewall:
- To generate an API key, you can use the following command:
-
panxapi.py -h PAN_MGMT_IP_OR_FQDN
-l USERNAME :'PASSWORD' -k
- (Recommended) store your API key in a .panrc file:
- As a good practice, avoid storing your API key directly in your script:
-
panxapi.py -h PAN_MGMT_IP_OR_FQDN
-l USERNAME :'PASSWORD' -k >> ~/.panrc
Other Considerations
If this linux instance is not behind the same public IP that the FQDN will resolve to, you may need to create a NAT rule on your firewall. Certbot assumes that the certificate will be installed on the host issuing the call. While most linux based web servers make this process easy, network devices typically do not.
- If your linux instance is not behind the same public IP as your VPN Portal/Gateway, you can create a NAT rule to ensure LetsEncrypt “sees” this host coming from the same public IP.
- If your instance is NAT’d to the same public IP that your GlobalProtect Portal/Gateway uses, you can skip this step
- If you are using a dns-provider plugin that handles domain validation for you, you can skip this step.
Initial Configuration - CloudFlare
To setup certbot initially, issue the following command. This command will generate certificates non-interactively, automatically running a standalone web server for authentication and accepting the ToS. While we can certainly generate and/or renew interactively, the ultimate goal is unattended automation.
- Replace
*.bitbodyguard.com with the desired certificate FQDN or a comma-separated list of domains. - Note: The first domain provided will be the subject CN on the certificate! All other domains will appear as SANs
/home/psiri/.cloudflare.ini” with the full path to your stored CloudFlare CredentialsReplace “ In this example, I am requesting a wildcard certificate, so I will use “*.bitbodyguard.com”
cloudflare-credentials /home/psiri/.cloudflare.ini -d *.bitbodyguard.com --preferred-challenges dns-01
Initial Configuration - Standalone
If you are not using CloudFlare DNS for authentication, you can optionally use the standalone method to perform initial certbot setup:
- Replace *.bitbodyguard.com
with the desired certificate FQDN or a comma-separated list of domains. - Note: The first domain provided will be the subject CN on the certificate! All other domains will appear as SANs
In this example, I am requesting a wildcard certificate, so I will use “*.bitbodyguard.com” - Replace <USERNAME>
@<YOUR-DOMAIN> with your email address. This is used for important account notifications.
certbot certonly -d *.bitbodyguard.com -m <USERNAME>@<YOUR-DOMAIN> --standalone -n --agree-tos
The Script
You can either copy the code below, or clone directly from GitHub
#!/bin/bash CLOUDFLARE_CREDS=/home/psiri/.cloudflare.ini PAN_MGMT=10.0.100.2 FQDN=bitbodyguard.com [email protected] API_KEY=$(cat /home/psiri/.panrc) CERT_NAME=LetsEncryptWildcard GP_PORTAL_TLS_PROFILE=GP_PORTAL_PROFILE GP_GW_TLS_PROFILE=GP_EXT_GW_PROFILE TEMP_PWD=$(openssl rand -hex 15) #Requirements: openssl, pan-python, certbot sudo /usr/local/bin/certbot certonly --dns-cloudflare --dns-cloudflare-credentials $CLOUDFLARE_CREDS -d *.$FQDN -n --agree-tos --force-renew #Depending on your setup, certbot may not give you separate files for the certificate and chain. This script expects separate files. sudo openssl pkcs12 -export -out letsencrypt_pkcs12.pfx -inkey /etc/letsencrypt/live/$FQDN/privkey.pem -in /etc/letsencrypt/live/$FQDN/cert.pem -certfile /etc/letsencrypt/live/$FQDN/chain.pem -passout pass:$TEMP_PWD curl -k --form file=@letsencrypt_pkcs12.pfx "https://$PAN_MGMT/api/?type=import&category=certificate&certificate-name=$CERT_NAME&format=pkcs12&passphrase=$TEMP_PWD&key=$API_KEY" && echo " " curl -k --form file=@letsencrypt_pkcs12.pfx "https://$PAN_MGMT/api/?type=import&category=private-key&certificate-name=$CERT_NAME&format=pkcs12&passphrase=$TEMP_PWD&key=$API_KEY" && echo " " sudo rm letsencrypt_pkcs12.pfx #If you use a separate SSL/TLS Service Profile for the GlobalProtect Portal and Gateway, uncomment the next line and update the 'GP_PORTAL_TLS_PROFILE' variable with the name of your GlobalProtect Portal's SSL/TLS Service Profile, as it appears in your management GUI. panxapi.py -h $PAN_MGMT -K $API_KEY -S "$CERT_NAME " "/config/shared/ssl-tls-service-profile/entry[@name='$GP_PORTAL_TLS_PROFILE']" #If you use a separate SSL/TLS Service Profile for the GlobalProtect Portal and Gateway, uncomment the next line and update the 'GP_GW_TLS_PROFILE' variable with the name of your GlobalProtect Gateway's SSL/TLS Service Profile, as it appears in your management GUI. If you use a single SSL/TLS Service Profile for BOTH the Portal and Gateway, you can comment the following line out, or set the value of 'GP_GW_TLS_PROFILE' to the value of 'GP_PORTAL_TLS_PROFILE' panxapi.py -h $PAN_MGMT -K $API_KEY -S "$CERT_NAME " "/config/shared/ssl-tls-service-profile/entry[@name='$GP_GW_TLS_PROFILE']" panxapi.py -h $PAN_MGMT -K $API_KEY -C '' --sync
How It Works
This small bash script performs the following actions:
- Uses the provided CloudFlare credentials file to call certbot’s dns-cloudflare plugin, automatically validating ownership of the FQDN/domain by creating and deleting TXT records
- Accepts LetsEncrypt’s ToS and renews the certificate(s) for the provided FQDN(s)
- Randomly generates a certificate passphrase using “openssl rand”
- Creates a temporary, password-protected PKCS12 cert file named “letsencrypt_pkcs12.pfx” from the individual private and public keys issued by LetsEncrypt.
- Uploads the temporary PKCS12 file to the firewall using the randomly-generated passphrase. Certificate name is set to variable $CERT_NAME
- Deletes the temporary PKCS12 certificate from linux host
- (Optionally) Sets the certificate used within the GlobalProtect Portal’s SSL/TLS profile to the name of the new LetsEncrypt certificate
- (Optionally) Sets the certificate used within the GlobalProtect Gateway’s SSL/TLS profile to the name of the new LetsEncrypt certificate
- Commits the candidate configuration (synchronously) and reports for the commit result
Automated Renewal and Installation
At this point, we have everything we need to put our setup into a cronjob which will automatically renew and upload the certificates, modify the SSL/TLS Service Profiles (if required), and commit the configuration.
To make this a bit more adaptable to different scenarios, I have included the following variables:
- CLOUDFLARE_CREDS: The full path to the “.cloudflare.ini” file
- CERT_NAME: The name you wish to give the certificate on the device (Palo Alto Networks GUI: Device –> Certificate Management –> Certificates)
- GP_PORTAL_TLS_PROFILE: The name of the GlobalProtect SSL/TLS Service Profile used on the Portal.
- GP_GW_TLS_PROFILE: The name of the GlobalProtect SSL/TLS Service Profile used on the Gateway. For single Portal/Gateway deployments using a single SSL/TLS profile, this may be the same as “GP_PORTAL_TLS_PROFILE”.
![SSL/TLS Service Profiles](https://storage.googleapis.com/webmedia.gcp.bitbodyguard.com/Web%20Media/Page_LetsEncrypt_PaloAlto/SSL-TLS_Profiles.png)
NOTES:
- As best-practice, you should use separate SSL/TLS Service Profiles for each Portal and Gateway. This script assumes you have followed best-practices, but will also work with single-profile configurations.
- With Palo Alto Networks Firewalls specifically, updating the SSL/TLS Service Profiles is only required when the name of the certificate referenced by the SSL/TLS Service Profile changes.
- If the SSL/TLS Service Profiles have been updated with the name of the LetsEncrypt certificate (and you do not plan on timestamping or otherwise changing the certificate name), no modifications are required when a certificate is renewed. If you choose to append a timestamp or rename the certificate, you will need to programmatically update the SSL/TLS Service profiles.
- To update the SSL/TLS Service Profile(s), uncomment lines 20 (for a single SSL/TLS Profile) and 22 (for two SSL/TLS Profiles)
-
panxapi.py -h $PAN_MGMT -K $API_KEY -S "
$CERT_NAME " "/config/shared/ssl-tls-service-profile/entry[@name='$GP_PORTAL_TLS_PROFILE']"
panxapi.py -h $PAN_MGMT -K $API_KEY -S "$CERT_NAME " "/config/shared/ssl-tls-service-profile/entry[@name='$GP_GW_TLS_PROFILE']" - If your certificate name does not change, you can safely leave these commands enabled if desired
Prepare the Script for Execution
- If the cron executes as another logged-in user, you may need to change relative paths to full-paths. (Note – This has been done already above)
- Ex: “~/.panrc” to “/home/<username>/.panrc”
- Or: “~/.cloudflare.ini” to /home/<username>/.cloudflare.ini
- Ensure there are no extensions after the filename. In most cases, only underscores and dashes are supported
- Ensure the cron is executable
-
sudo chmod +x /home/psiri/pan_certbot
-
Creating a Cronjob for Fully-Automated Certificate Installation
Note: This article is written for Ubuntu
Create a custom cronjob to execute the certificate renewal at your desired interval:
-
sudo vi /etc/cron.d/pan_certbot_cron
- Insert the following within the /etc/cron.d/pan_certbot file to ensure the cron generates a log file to debug any issues:
- The following cronjob will run twice weekly (Midnight Weds and Saturday). To change execution frequency, modify the schedule to meet your needs.
- The log file will be timestamped and stored in /var/log/pan_certbot.log
- Change “/home/psiri/pan_certbot” to the full path of the script
-
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHELL=/bin/bash 0 0 * * 3,6 root (/bin/date && /home/psiri/pan_certbot) >> /var/log/pan_certbot.log 2>&1
LetsEncrypt Renewal Limits
LetsEncrypt rate-limits the renewal of certificates by default. It is not uncommon to hit the main limit – 50 certificates per registered domain, per week. Certificate renewals also have a special “Duplicate Certificate” limit of 5/week which you are likely to hit with frequently-running jobs.
There is no penalty for exceeding these limits. If you have a limited number of certificates to renew, weekly or bi-weekly cronjobs will work perfectly. If you’re running this as a daily or hourly cronjob, you will inevitably see the following error:
Renewing an existing certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: *.bitbodyguard.com: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details.
Validate Your Runs
cat /var/log/pan_certbot.log
If successful, you should see something similar to the following:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Renewing an existing certificate
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/bitbodyguard.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/bitbodyguard.com/privkey.pem
Your cert will expire on 2020-01-21. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4536 100 125 100 4411 34 1204 0:00:03 0:00:03 --:--:-- 1204
<response status="success"><result>Successfully imported LetsEncryptWildcard into candidate configuration</result></response>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4536 100 125 100 4411 103 3667 0:00:01 0:00:01 --:--:-- 3666
<response status="success"><result>Successfully imported LetsEncryptWildcard into candidate configuration</result></response>
set: success [code="20"]: "command succeeded"
set: success [code="20"]: "command succeeded"
commit: success: "Configuration committed successfully"
If the cron was successful, but you have it a rate-limit, you will typically see the following:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-cloudflare, Installer None
Renewing an existing certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: *.bitbodyguard.com: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4536 100 125 100 4411 101 3582 0:00:01 0:00:01 --:--:-- 3583
Successfully imported LetsEncryptWildcard into candidate configuration
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4536 100 125 100 4411 160 5678 --:--:-- --:--:-- --:--:-- 5676
Successfully imported LetsEncryptWildcard into candidate configuration
set: success [code="20"]: "command succeeded"
set: success [code="20"]: "command succeeded"
commit: success: "Configuration committed successfully"
You should also be able to check the following locations on the Palo Alto Networks firewall for additional confirmation:
Device –> Certificate Management –> Certificates
Monitor –> Logs –> Configuration
You should see 3-5 operations, depending on whether or not you chose to modify the SSL/TLS service profile(s). In my setup (PA-850), it takes 7-8 seconds to renew, upload, and commit the configuration (not including actual commit time):
- A web upload to /config/shared/certificate
- A web upload to /config/shared/certificate/entry[@name=’LetsEncryptWildcard’]
- A web “set” command to /config/shared/ssl-tls-service-profile/entry[@name=’GP_PORTAL_PROFILE’]
- A web “set” command to /config/shared/ssl-tls-service-profile/entry[@name=’GP_EXT_GW_PROFILE’]
- And a web “commit” operation