MISP/Cuckoo

Quick Integration of MISP and Cuckoo

With the number of attacks that we are facing today, defenders are looking for more and more IOC’s (“Indicator of Compromise) to feed their security solutions (firewalls, IDS, …). It becomes impossible to manage all those IOC’s manually and automation is the key. There are two main problems with this amount of data:

  1. How to share them in a proper way (remember: sharing is key).
  2. How to collect and prepare them to be shared.

Note that I’m considering in this post only the “technical” issues with IOC’s. There are much more issues like their accuracy (which can be different between different environments).

To search for IOC’s, I’m using the following environment: A bunch of honeypots capture samples that, if interesting, are analyzed by a Cuckoo sandbox. To share the results with peers, a MISP instance is used.

MISP-Cuckoo

In this case, a proper integration between Cuckoo and MISP is the key. It is implemented in both ways. The results of the Cucko analyzis are enriched with IOC’s found in MISP. IOC’s found in the sample are correlated with MISP and the event ID, description and level are displayed:

In the other way, Cuckoo submits the results of the ianalyzes to MISP:

Cuckoo 2.0 comes with ready-to-use modules to interact with the MISP REST API via the PyMISP Python module. There is one processing module (to search for existing IoC’s in MISP) and one reporting module (to create a new event in MISP). The configuration is very simple, just define your MISP URL and API key in the proper configuration files and you’re good to go:

# cd $CUCKOO_HOME/conf
# grep -A 2 -B 2 misp *.conf
processing.conf-enabled = yes
processing.conf-
processing.conf:[misp]
processing.conf-enabled = yes
processing.conf:url = https://misp.xxxxxxxxxx
processing.conf-apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
maxioc = 100
--
reporting.conf-logname = syslog.log
reporting.conf-
reporting.conf:[misp]
reporting.conf-enabled = yes
reporting.conf:url = https://misp.xxxxxxxxxx
reporting.conf-apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

But, in my environment, the default modules generated too many false positives in both ways. I patched them to support the more configuration keywords to better control what is exchanged between the two tools.

In the processing module:

only_ids = [yes|no]

If defined, only attributes with the “IDS” flag set to “1” will be displayed in Cuckoo.

ioc_blacklist = 8.8.8.8,8.8.4.4,www.google.com

This parameter allows you to define a list (comma delimited) of IOC’s that you do NOT want in Cuckoo. Typically, you’ll add here DNS servers, specific URLs.

In the reporting module:

tag = Cuckoo

Specify the tag to be added to created event in MISP. Note that the tag must be created manually.

# Default distribution level:
# your_organization = 0
# this_community = 1
# connected_communities = 2
# all_communities = 3
distribution=0

The default distribution level assigned to the created event (default: 0)

# Default threat level:
# high = 1
# medium = 2
# low = 3
# undefined = 4
threat_level_id=4

The default threat level assigned to the created event (default: 4)

# Default analysis level:
# initial = 0
# ongoing = 1
# completed = 2
analysis = 0

The default analysis status assigned to the created event (default: 0)

ioc_blacklist = 8.8.8.8,8.8.4.4,www.google.com,crl.verisign.com,sc.symcb.com

Again, a blacklist (comma separated) of IOC’s that you do NOT want to store in MISP.

Events are created in MISP with an “unpublished” status. This allows you to review them, add manually some IOC’s, to merge different events, add some tags or change default values.

The patched files for Cuckoo are available in my GitHub repository.

16 comments

  1. Any information about this yet ?

    “Error runing cuckoo after replacing py misp files for the processing module files aka misp.py and processing.conf.
    Any tips how to fix this ?

    [sudo] password for cuckoo:
    Traceback (most recent call last):
    File “/usr/local/bin/cuckoo”, line 11, in
    load_entry_point(‘Cuckoo==2.0.3’, ‘console_scripts’, ‘cuckoo’)()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 560, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2648, in load_entry_point
    return ep.load()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2302, in load
    return self.resolve()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2308, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/__init__.py”, line 6, in
    from cuckoo import (
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/__init__.py”, line 10, in
    __file__, “cuckoo.processing”, globals(), Processing
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/core/plugins.py”, line 39, in enumerate_plugins
    “%s.%s” % (module_prefix, module_name)
    File “/usr/lib/python2.7/importlib/__init__.py”, line 37, in import_module
    __import__(name)
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/misp.py”, line 54
    else:
    ^
    IndentationError: expected an indented block

  2. Issue solved.
    The problem was about flow difference to reach MISP (internal, no proxy) & VT (external, proxy) by Cuckoo.

    To allow this, I used the “no_proxy” variable in the bash (persistant).
    The syntax is strict: no CIDR and no wildcard.
    export no_proxy=”127.0.0.1,localhost,10.1.2.3,.myciedomain.com”

    Now it would works.
    Regards,

  3. Error runing cuckoo after replacing py misp files for the processing module files aka misp.py and processing.conf.
    Any tips how to fix this ?

    [sudo] password for cuckoo:
    Traceback (most recent call last):
    File “/usr/local/bin/cuckoo”, line 11, in
    load_entry_point(‘Cuckoo==2.0.3’, ‘console_scripts’, ‘cuckoo’)()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 560, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2648, in load_entry_point
    return ep.load()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2302, in load
    return self.resolve()
    File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2308, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/__init__.py”, line 6, in
    from cuckoo import (
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/__init__.py”, line 10, in
    __file__, “cuckoo.processing”, globals(), Processing
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/core/plugins.py”, line 39, in enumerate_plugins
    “%s.%s” % (module_prefix, module_name)
    File “/usr/lib/python2.7/importlib/__init__.py”, line 37, in import_module
    __import__(name)
    File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/misp.py”, line 54
    else:
    ^
    IndentationError: expected an indented block

  4. Hi xme,

    this is EXACTLY what I need to achive with my own Cuckoo and MISP setup to “filter” out my own malware analysis from the foreign IOC’s.
    But using the spender modified cuckoo, it does not work adding the “tag = cuckoo” into the reporting.conf.
    You wrote, that you have patched your models to accomplish the task. I’ve checked the links provided, but they come up wth a 404.
    So could you make your patching (or the modules) public or share me a copy?

    Thanks a lot
    Marcus

  5. That’s… interesting! I’ve no idea about this issue… Is the cuckoo.conf correct? (like a non printable character in the config?)

  6. Very strange.
    My API has been renewed.
    HTTPS has been removed.
    From Cuckoo box, I ran an example PyMISP script (last.py) and it can retrieve some IOCs ; with the same inputs (UR & APIkey).

    But it seems that Cuckoo can’t reach MISP during analysis process and I get the same error.
    Here is the full log on Cuckoo side:
    ====
    2017-02-23 11:27:43,677 [lib.cuckoo.core.plugins] ERROR: Failed to run the processing module “MISP” for task #87:
    Traceback (most recent call last):
    File “/opt/cuckoo/lib/cuckoo/core/plugins.py”, line 242, in process
    data = current.run()
    File “/opt/cuckoo/modules/processing/misp.py”, line 81, in run
    self.misp = PyMISP(self.url, self.apikey, False, “json”)
    File “/usr/lib/python2.7/site-packages/pymisp/api.py”, line 113, in __init__
    raise PyMISPError(‘Unable to connect to MISP ({}). Please make sure the API key and the URL are correct (http/https is required): {}’.format(self.root_url, e))
    PyMISPError: Unable to connect to MISP (http:///). Please make sure the API key and the URL are correct (http/https is required): 504 Server Error: Gateway Timeout
    ===
    Here is the full log on MISP side. I get some 403 errors, but I didn’t found this file on server so I couldn’t check permissions.
    ===
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “POST /events/restSearch/download HTTP/1.1” 403 299 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:27 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:27 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:28 +0100] “POST /events/restSearch/download HTTP/1.1” 403 299 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:13:17 +0100] “POST /events/restSearch/download HTTP/1.1” 200 910165 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 200 20 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 200 15178 “-” “PyMISP 2.4.65 – Python 2.7.5”
    xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “POST /events/restSearch/download HTTP/1.1” 200 910165 “-” “PyMISP 2.4.65 – Python 2.7.5”

    I think I will open a issue on MISP’s Github.
    Thanks,
    Regards,

  7. Many thanks Xavier for your post.
    It’s very useful if you begin with MISP 🙂

    Tried today, but after reviewing my settings, I still reach an error on Cuckoo’s side when I submit a sample:
    PyMISPError: Unable to connect to MISP (http://<my_misp_IP/). Please make sure the API key and the URL are correct (http/https is required): 504 Server Error: Gateway Timeout

    If you have any idea 😉
    Thanks again.

Leave a Reply

Your email address will not be published. Required fields are marked *