Managing Palo Alto Firewalls Custom URL Categories

Palo Alto Networks firewalls are very popular due to the huge amount of features they provide in a unique chassis. Besides the traditional traffic inspection, they can play up to the 7th layer of the ISO model. The rule base can contain rules which inspect the web traffic and prevent users to access specific URLs. The classic model used is the URLs categorization in multiple topics like the classic “Games“, “Adult“, “Auctions“, “Hacking“, etc. (A complete list of categories is available here). But they also provide a way to create custom categories that can be populated later. There is a feature called “Dynamic Block Lists” that allows the creation of objects which will be populated automatically with data pulled over HTTP (like a crontab). But such lists are only based on IP addresses.

I had to do the same for URLs and I wrote a small Python script “pum.py” (which stands for “Palo Alto URL Manager“) to manage dynamic lists of URLs. It helps to add, delete and list URLs stored on multiple firewalls and customer URL categories. The purpose of the script is to be called from other applications/scripts. In my project, it is called from a Splunk instance. The syntax is easy to understand:

$ ./pum.py -h
usage: pum.py [-h] [-f FIREWALL [FIREWALL ...]] [-c CONFIGFILE]
 [-u URLCATEGORY] [-n] [-v] [-s] [-p] [-l | -a | -d]
 [URL [URL ...]]

Manage custom URL categories in a Palo Alto Networks firewall

positional arguments:
 URL the URL(s) to add/remove (format is IP/FQDN)

optional arguments:
 -h, --help show this help message and exit
 -f FIREWALL [FIREWALL ...], --firewall FIREWALL [FIREWALL ...]
 firewall IP address or hostname
 -c CONFIGFILE, --config CONFIGFILE
 configuration file (default: /etc/pum.conf)
 -u URLCATEGORY, --url-category URLCATEGORY
 custom URL category to manage
 -n, --no-ssl-verification
 ignore SSL certificate errors
 -v, --verbose enable verbose output
 -s, --save commit the configuration (save)
 -p, --partial perform a partial commit (objects only)
 -l, --list list URLs from the custom URL category
 -a, --add add the URL(s) to the custom URL category
 -d, --delete remove the URL(s) from the custom URL category

The script uses the Palo Alto API to talk to the firewalls. You just need to create an API key and store it in a configuration file. You can define as many firewall as you have:

$ cat pum.conf
[192.168.0.1]
apikey: <redacted>
urlcategory: my_malicious_urls

[192.168.0.2]
apikey: <redacted>
urlcategory: my_malicious_urls

# To be added soon...
[proxy]
host:
port:
username:
password:

To add an URL, use the following command:

$ ./pum.py -v -f 192.168.1.1 -c pum.conf -n -a www.maliciousdomain.cn
+++ Verbose output enabled
+++ Processing firewall 192.168.1.1
+++ WARNING: SSL certificate check disabled
+++ Added www.maliciousdomain.cn in my_malicious_urls

It is possible to record multiple URLs at a time by reading them from STDIN:

$ cat bad_urls.txt
www.baddomain2.ru
c2.malicious.com
www.pwnedwordpress.fr
$ cat bad_urls.txt | ./pum.py -f 192.168.1.1 -c pum.conf -v -n -a -
+++ Verbose output enabled
+++ Processing firewall 192.168.1.1
+++ WARNING: SSL certificate check disabled
+++ Added www.baddomain2.ru in my_malicious_urls
+++ Added c2.malicious.com in my_malicious_urls
+++ Added www.pwnedwordpress.fr in my_malicious_urls

It is possible to list the URLs present in a specific category:

$ ./pum.py -v -f 192.168.1.1 -c pum.conf -n -l
+++ Verbose output enabled
+++ Processing firewall 192.168.1.1
+++ WARNING: SSL certificate check disabled
+++ Fetching list of URLs for 192.168.1.1/my_malicious_urls
2015/12/23 09:42:52 | 192.168.1.1 | my_malicious_urls | www.maliciousdomain.cn
2015/12/23 09:44:38 | 192.168.1.1 | my_malicious_urls | www.baddomain2.ru
2015/12/23 09:44:38 | 192.168.1.1 | my_malicious_urls | c2.malicious.com
2015/12/23 09:44:38 | 192.168.1.1 | my_malicious_urls | www.pwnedwordpress.fr

The custom URL category is populated and available immediately in the web GUI:

And can be applied to rules in your security policy:

Some useful tips:

• The customer URL category must exists in the firewall configuration
• By default, the new policy is not committed (use the flag ‘-s’ to commit and ‘-p’ to make a partial commit)
• The script has been tested against PanOS 7.x (I know that the API syntax changes sometimes)

The script is available here.