With the number of attacks that we are facing today, defenders are looking for more and more IOC’s (“Indicator of Compromise) to feed their security solutions (firewalls, IDS, …). It becomes impossible to manage all those IOC’s manually and automation is the key. There are two main problems with this amount of data:
- How to share them in a proper way (remember: sharing is key).
- How to collect and prepare them to be shared.
Note that I’m considering in this post only the “technical” issues with IOC’s. There are much more issues like their accuracy (which can be different between different environments).
To search for IOC’s, I’m using the following environment: A bunch of honeypots capture samples that, if interesting, are analyzed by a Cuckoo sandbox. To share the results with peers, a MISP instance is used.
In this case, a proper integration between Cuckoo and MISP is the key. It is implemented in both ways. The results of the Cucko analyzis are enriched with IOC’s found in MISP. IOC’s found in the sample are correlated with MISP and the event ID, description and level are displayed:
In the other way, Cuckoo submits the results of the ianalyzes to MISP:
Cuckoo 2.0 comes with ready-to-use modules to interact with the MISP REST API via the PyMISP Python module. There is one processing module (to search for existing IoC’s in MISP) and one reporting module (to create a new event in MISP). The configuration is very simple, just define your MISP URL and API key in the proper configuration files and you’re good to go:
# cd $CUCKOO_HOME/conf # grep -A 2 -B 2Â misp *.conf processing.conf-enabled = yes processing.conf- processing.conf:[misp] processing.conf-enabled = yes processing.conf:url = https://misp.xxxxxxxxxx processing.conf-apikey =Â xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx maxioc = 100 -- reporting.conf-logname = syslog.log reporting.conf- reporting.conf:[misp] reporting.conf-enabled = yes reporting.conf:url = https://misp.xxxxxxxxxx reporting.conf-apikey =Â xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
But, in my environment, the default modules generated too many false positives in both ways. I patched them to support the more configuration keywords to better control what is exchanged between the two tools.
In the processing module:
only_ids = [yes|no]
If defined, only attributes with the “IDS” flag set to “1” will be displayed in Cuckoo.
ioc_blacklist = 8.8.8.8,8.8.4.4,www.google.com
This parameter allows you to define a list (comma delimited) of IOC’s that you do NOT want in Cuckoo. Typically, you’ll add here DNS servers, specific URLs.
In the reporting module:
tag = Cuckoo
Specify the tag to be added to created event in MISP. Note that the tag must be created manually.
# Default distribution level: # your_organization = 0 # this_community = 1 # connected_communities = 2 # all_communities = 3 distribution=0
The default distribution level assigned to the created event (default: 0)
# Default threat level: # high = 1 # medium = 2 # low = 3 # undefined = 4 threat_level_id=4
The default threat level assigned to the created event (default: 4)
# Default analysis level: # initial = 0 # ongoing = 1 # completed = 2 analysis = 0
The default analysis status assigned to the created event (default: 0)
ioc_blacklist = 8.8.8.8,8.8.4.4,www.google.com,crl.verisign.com,sc.symcb.com
Again, a blacklist (comma separated) of IOC’s that you do NOT want to store in MISP.
Events are created in MISP with an “unpublished” status. This allows you to review them, add manually some IOC’s, to merge different events, add some tags or change default values.
The patched files for Cuckoo are available in my GitHub repository.
Hi,
I am trapped in the following error. Does these mean anything to you. I have patched both .py files as well.
Yep,
Do you know if the configuration syntax changed in the latest version: 2.0.5.3?
At Cuckoo restart, it raises some errors:
2018-02-16 16:28:27,933 [cuckoo.common.config] ERROR: Type of config parameter processing:misp:only_ids not found! This may indicate that you’ve incorrectly filled out the Cuckoo configuration, please double check it.
2018-02-16 16:28:27,933 [cuckoo.common.config] ERROR: Type of config parameter processing:misp:ioc_blacklist not found! This may indicate that you’ve incorrectly filled out the Cuckoo configuration, please double check it.
2018-02-16 16:28:27,939 [cuckoo.common.config] ERROR: Type of config parameter reporting:misp:min_malscore not found! This may indicate that you’ve incorrectly filled out the Cuckoo configuration, please double check it.
2018-02-16 16:28:27,940 [cuckoo.common.config] ERROR: Type of config parameter reporting:misp:threads not found! This may indicate that you’ve incorrectly filled out the Cuckoo configuration, please double check it.
Thanks,
Julien
Any information about this yet ?
“Error runing cuckoo after replacing py misp files for the processing module files aka misp.py and processing.conf.
Any tips how to fix this ?
[sudo] password for cuckoo:
Traceback (most recent call last):
File “/usr/local/bin/cuckooâ€, line 11, in
load_entry_point(‘Cuckoo==2.0.3’, ‘console_scripts’, ‘cuckoo’)()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.pyâ€, line 560, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.pyâ€, line 2648, in load_entry_point
return ep.load()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.pyâ€, line 2302, in load
return self.resolve()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.pyâ€, line 2308, in resolve
module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
File “/usr/local/lib/python2.7/dist-packages/cuckoo/__init__.pyâ€, line 6, in
from cuckoo import (
File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/__init__.pyâ€, line 10, in
__file__, “cuckoo.processingâ€, globals(), Processing
File “/usr/local/lib/python2.7/dist-packages/cuckoo/core/plugins.pyâ€, line 39, in enumerate_plugins
“%s.%s†% (module_prefix, module_name)
File “/usr/lib/python2.7/importlib/__init__.pyâ€, line 37, in import_module
__import__(name)
File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/misp.pyâ€, line 54
else:
^
IndentationError: expected an indented block
“
Issue solved.
The problem was about flow difference to reach MISP (internal, no proxy) & VT (external, proxy) by Cuckoo.
To allow this, I used the “no_proxy” variable in the bash (persistant).
The syntax is strict: no CIDR and no wildcard.
export no_proxy=”127.0.0.1,localhost,10.1.2.3,.myciedomain.com”
Now it would works.
Regards,
Error runing cuckoo after replacing py misp files for the processing module files aka misp.py and processing.conf.
Any tips how to fix this ?
[sudo] password for cuckoo:
Traceback (most recent call last):
File “/usr/local/bin/cuckoo”, line 11, in
load_entry_point(‘Cuckoo==2.0.3’, ‘console_scripts’, ‘cuckoo’)()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 560, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2648, in load_entry_point
return ep.load()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2302, in load
return self.resolve()
File “/home/cuckoo/.local/lib/python2.7/site-packages/pkg_resources/__init__.py”, line 2308, in resolve
module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
File “/usr/local/lib/python2.7/dist-packages/cuckoo/__init__.py”, line 6, in
from cuckoo import (
File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/__init__.py”, line 10, in
__file__, “cuckoo.processing”, globals(), Processing
File “/usr/local/lib/python2.7/dist-packages/cuckoo/core/plugins.py”, line 39, in enumerate_plugins
“%s.%s” % (module_prefix, module_name)
File “/usr/lib/python2.7/importlib/__init__.py”, line 37, in import_module
__import__(name)
File “/usr/local/lib/python2.7/dist-packages/cuckoo/processing/misp.py”, line 54
else:
^
IndentationError: expected an indented block
It’s online: https://github.com/xme/cuckoo
Hi xme,
this is EXACTLY what I need to achive with my own Cuckoo and MISP setup to “filter” out my own malware analysis from the foreign IOC’s.
But using the spender modified cuckoo, it does not work adding the “tag = cuckoo” into the reporting.conf.
You wrote, that you have patched your models to accomplish the task. I’ve checked the links provided, but they come up wth a 404.
So could you make your patching (or the modules) public or share me a copy?
Thanks a lot
Marcus
Yes, it is.
It seems this issue come from http_proxy environment used to allow VT outbound connection.
Cuckoo community invites to use /etc/hosts entry to do it.
I must read info to do it.
* https://github.com/MISP/MISP/issues/1985
* https://github.com/cuckoosandbox/cuckoo/issues/1298
Thanks,
That’s… interesting! I’ve no idea about this issue… Is the cuckoo.conf correct? (like a non printable character in the config?)
Very strange.
My API has been renewed.
HTTPS has been removed.
From Cuckoo box, I ran an example PyMISP script (last.py) and it can retrieve some IOCs ; with the same inputs (UR & APIkey).
But it seems that Cuckoo can’t reach MISP during analysis process and I get the same error.
Here is the full log on Cuckoo side:
====
2017-02-23 11:27:43,677 [lib.cuckoo.core.plugins] ERROR: Failed to run the processing module “MISP” for task #87:
Traceback (most recent call last):
File “/opt/cuckoo/lib/cuckoo/core/plugins.py”, line 242, in process
data = current.run()
File “/opt/cuckoo/modules/processing/misp.py”, line 81, in run
self.misp = PyMISP(self.url, self.apikey, False, “json”)
File “/usr/lib/python2.7/site-packages/pymisp/api.py”, line 113, in __init__
raise PyMISPError(‘Unable to connect to MISP ({}). Please make sure the API key and the URL are correct (http/https is required): {}’.format(self.root_url, e))
PyMISPError: Unable to connect to MISP (http:///). Please make sure the API key and the URL are correct (http/https is required): 504 Server Error: Gateway Timeout
===
Here is the full log on MISP side. I get some 403 errors, but I didn’t found this file on server so I couldn’t check permissions.
===
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:23 +0100] “POST /events/restSearch/download HTTP/1.1” 403 299 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:27 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:27 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 403 301 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:08:28 +0100] “POST /events/restSearch/download HTTP/1.1” 403 299 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:13:17 +0100] “POST /events/restSearch/download HTTP/1.1” 200 910165 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “GET /servers/getPyMISPVersion.json HTTP/1.1” 200 20 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “GET /attributes/describeTypes.json HTTP/1.1” 200 15178 “-” “PyMISP 2.4.65 – Python 2.7.5”
xxx.xxx.xxx.xxx – – [23/Feb/2017:14:17:07 +0100] “POST /events/restSearch/download HTTP/1.1” 200 910165 “-” “PyMISP 2.4.65 – Python 2.7.5”
I think I will open a issue on MISP’s Github.
Thanks,
Regards,
Nothing related to the module… Low level issue: can you reach MISP from your Cuckoo box? API is ok?
Many thanks Xavier for your post.
It’s very useful if you begin with MISP 🙂
Tried today, but after reviewing my settings, I still reach an error on Cuckoo’s side when I submit a sample:
PyMISPError: Unable to connect to MISP (http://<my_misp_IP/). Please make sure the API key and the URL are correct (http/https is required): 504 Server Error: Gateway Timeout
If you have any idea 😉
Thanks again.