A Single Partner for Everything You Need Optiv works with more than 450 world-class security technology partners. By putting you at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can.
We Are Optiv Greatness is every team working toward a common goal. Winning in spite of cyber threats and overcoming challenges in spite of them. It’s building for a future that only you can create or simply coming home in time for dinner. However you define greatness, Optiv is in your corner. We manage cyber risk so you can secure your full potential.
Breadcrumb Home Insights Source Zero Sigma Rule Translation and Automatic Queries May 26, 2022 Sigma Rule Translation and Automatic Queries There are many different SIEM and EDR tools which, for the most part, use their own query language for information retrieval. When security team members cycle out of an organization their tribal knowledge and the specialized understanding of each query language is lost. Our industry is plagued by numerous query languages which hinders our collective ability to execute defenses effectively. But: what if queries could be written in one language and then translated into over 20 others? Such a development would reduce the learning curve to request specific information from various security tools and enhance detection efficacy. An emerging standardized language does exist: Sigma. Sigma is a generic rule format, written in YAML, to express search queries on log data. The project can be found here: https://github.com/SigmaHQ. How does this help and what does this mean? It means rules are formatted in a way that they can be translated between various query languages. As depicted in the diagram below, a Sigma rule can be translated to search/hunt across several SIEM and EDR platforms without the analyst having to know the query syntax of each platform. Image An example Sigma rule from the official GitHub project is shown below. Example: sysmon_accessing_winapi_in_powershell_credentials_dumping.yml title: Accessing WinAPI in PowerShell for Credentials Dumping id: 3f07b9d1-2082-4c56-9277-613a621983cc description: Detects Accessing to lsass.exe by Powershell status: experimental author: oscd.community, Natalia Shornikova date: 2020/10/06 modified: 2021/05/24 references: - https://speakerdeck.com/heirhabarov/hunting-for-powershell-abuse tags: - attack.credential_access - attack.t1003.001 logsource: product: windows service: sysmon detection: selection: EventID: - 8 - 10 SourceImage|endswith: '\powershell.exe' TargetImage|endswith: '\lsass.exe' condition: selection falsepositives: - Unknown level: high I found these resources helpful for additional information on creating Sigma rules: https://github.com/SigmaHQ/sigma/wiki/Rule-Creation-Guide https://syedhasan010.medium.com/defenders-toolkit-102-sigma-rules-4a623acb2036 https://www.youtube.com/watch?v=OheVuE9Ifhs Sigma has been around for a few years and there are various contributors to the official project, as well as additional projects that serve as repos for Sigma rules. One such repo I found is listed here: https://github.com/mdecrevoisier/SIGMA-detection-rules. I found the rule translation process and the ability to potentially automate queries across platforms intriguing. Specifically, how could the translation outputs be used as queries with little to no human interaction? To answer that, let’s begin with how rules can be translated. The Sigma project contains a Python script called sigmac. Sigmac is a tool that preforms the translation between Sigma rules and various security query formats using a YAML configuration file that map Sigma fields to each vendors’ query language. SOCPrime provides a web translation tool that can be found here: https://uncoder.io/. Image For a proof-of-concept I wanted to see if I could automate the use of sigmac to convert rules to a specific query format. Once translated, I wondered if I could automatically run a query against the security tool for any hits? My initial thoughts were to use Microsoft robotic process automation with uncoder.io to copy and paste rules or Power Automate (I’m a big fan of Power Automate) and sigmac to tackle the individual tasks necessary to perform the conversion and query. I feel like I’ve spent enough time highlighting Microsoft’s Automation tools in past posts that I wanted a new challenge. So, this idea would need to be distilled down to smaller steps to successfully execute my proof-of-concept. I decided Python would be my tool of choice for this task. Trying my hand at something new provides me with an appreciation for the complexities that are encountered in other technical roles. Breaking the original idea down into smaller tasks provided me with the outline below: Make the project shareable Clone the Sigma repo Convert a subset of Sigma rules to Devo queries Write the translated rules to separate files for later use Pass the converted queries to an API request To make the proof-of-concept sharable via Docker seemed an obvious choice. Docker is freely available, easy to share, widely used, and there is a convenient extension in VS Code in which I could prove all of this out. Within the Docker file the Sigma repo is cloned. The Docker file below uses a Python image, installs the necessary dependencies and runs several scripts, which I’ll walk through, to achieve the goal of automating the translation and API queries. Dockerfile FROM python:3.8-slim-buster # setup dependencies RUN apt-get update RUN apt-get install xz-utils RUN apt-get -y install curl git RUN pip install pyyaml RUN pip install requests RUN pip install python-dateutil WORKDIR /usr/src/app/ COPY . . RUN git clone https://github.com/SigmaHQ/sigma.git RUN cp devo-win.yml sigma/tools/config/ RUN mkdir Translations RUN python translate.py RUN mv Devo_* Translations RUN python main.py CMD ["sh"] With the dependencies installed, the container clones the Sigma repo. With the repo cloned the next task was to run sigmac (rule translation script) against all the files in a directory. The translate.py code below does this. translate.py # import required modules import cmd import os from re import sub rootdir = '/usr/src/app/sigma/rules/windows/wmi_event' cmd = './sigma/tools/sigmac -t devo -c sigma/tools/config/devo-win.yml ' for subdir, dirs, files in os.walk(rootdir): for file in files: ruleoutput = 'Devo_'+ file devorule= ruleoutput.replace(".yml","") cmdLine = cmd +subdir+'/'+file + ' -o '+ devorule os.system(cmdLine) For this proof-of-concept, the script defines a root directory. In this case, it is the path of the Sigma rules for windows wmi events, which is a sub-directory. Within the folder there are three Sigma rules. Sigmac will translate each of the three rules in this folder into a Devo query. The script will also run correctly if pointed at the Windows parent directory. If Windows is listed as the parent directory sigmac would then translate all rules in every subfolder below windows parent directory. Note, this can take some time. Image The script also creates a new file for each translated Devo rule, appending Devo_ to the original filename. One thing to note when the rules are translated with sigmac or uncode.io all attributes such as title, author, MITRE IDs, etc. are removed and you are left with the raw query. After translation is complete, each new file is moved into the Translations folder. This action is defined in the docker file, line 19. Taking a look at the contents of the Translations folder we can see that there are three new files beginning with “Devo_”. Image The screenshot above also shows the contents of the file Devo_sysmon_wmi_event_subscription. Here we see a properly translated Devo query that aligns with original Sigma rule shown below. Image One item to note is that the proper translation of Sigma rules to Devo_windows queries are not working correctly without a little finessing of the devo_windows.yml config file. I have opened and issue regarding this: https://github.com/SigmaHQ/sigma/issues/2941. The issue has to do with a single field on line 125. Changing the line to Type:eventType as shown in the second screen shot below seemed to work for me. The edited file can be found here. Image Image: devo-windows.yml from https://github.com/SigmaHQ/sigma/blob/master/tools/config/devo-windows.yml Image Image: Corrected line Note: There is a devo-win.yml file on my github project that corrects this. After the Sigma rules were properly translated, the last step was to send an API query request to Devo. Devo does have a SDK, but I wanted to make the request without using it, so I created the devo_request.py script listed below. The script requires API keys for authentication. Once the keys are provided, the only argument required for the script to run correctly is a properly formatted Devo query. Devo API queries require that a start date/time and end date/time are defined as a string. The script automatically calculates a thirty-day window, in the proper format, from the time of script execution and will search over the last thirty days for whatever query it is provided. The devo_request.py is used to run individual queries. This means I would need to call the script as many times as there are new Devo translated queries within the Translations folder. devo_request.py from inspect import signature import time import datetime import dateutil.relativedelta import hmac import hashlib import json import requests import sys url = "https://apiv2-us.devo.com/search/query" #note: hard coding keys is a really bad practice and this is just poc code. api_key = 'YOUR-KEY' api_secret = 'YOU-SECRET' query = sys.argv[1] curr_dt = datetime.datetime.now() now = int(round(curr_dt.timestamp())) start_dt = (curr_dt + dateutil.relativedelta.relativedelta(days=-30)) start = int(round(start_dt.timestamp())) payload = json.dumps({ "query": query, "from": start, "to": now, "limit": 10, "mode": { "type": "json/compact" } }) a = bytes(api_secret, 'utf-8') timestamp = str(int(time.time()) * 1000) data = payload message = api_key + data + timestamp b = bytes(message, 'utf-8') sign = hmac.new(a, b, digestmod=hashlib.sha256).hexdigest() headers = { 'Content-Type': 'application/json', 'x-logtrust-apikey': api_key, 'x-logtrust-sign': sign, 'x-logtrust-timestamp': timestamp } response = requests.request("POST", url, data=payload, headers=headers) #print(response.text) #print(response.json()) json_object = json.dumps(response.json(), indent = 4) with open('response.txt', 'a') as outfile: outfile.write("\n"+ "Query results from the last 30 Days for query " + query + json_object +"\n") The request script can write the contents to screen or it can write the query results, in json format, to a file, response.txt. Each query result will be appended to the file so past queries are not overwritten. I could have used something like pandas to present the results to the user in a better format, but that was above and beyond my original goal. With the devo_request.py script written, I needed to find a way to open all the files within a folder, read the contents of each and pass the contents as the query argument in the devo_request.py script. The main.py script below does just that. With the Translations folder defined, the program walks the folder, opens each file, reads the contents and passes the text as the query parameter in the devo_request.py script. main.py from pathlib import Path import sys import cmd import os from re import sub import cmd cmd = 'python devo_request.py ' folder = "/usr/src/app/Translations/" for file in os.listdir(folder): filepath = os.path.join(folder, file) f = open(filepath, 'r') r = f.read() cmdLine = cmd +("\'"+r+"\'") print('\n') print("Query results from the last 30 Days for "+ r) f.close() os.system(cmdLine) Opening the response.txt file shows the exact Devo API query made, as well results or response of each query. This can be seen in the partial screenshot of the response.txt file. Image Stepping back from the code the entire flow looks like this: Image The docker image will drop you into a shell so you can poke around. There is a ton of room to expand on these ideas and improve my scripts. I would encourage anyone interested in content engineering or API queries to take a look. Tools referenced in this blog https://github.com/SigmaHQ https://github.com/SigmaHQ/sigma/wiki/Converter-Tool-Sigmac https://github.com/kiraly15/sigma_blog By: Dan Kiraly Senior Research Scientist | Optiv Dan Kiraly is senior research scientist on Optiv’s R&D team. In this role he's responsible for use case development and the vetting of security products for Optiv. Share: Source Zero® Blue Team Sigma rule translation automatic queries Docker YAML SIEM EDR query language log data Optiv Security: Secure greatness.® Optiv is the cyber advisory and solutions leader, delivering strategic and technical expertise to nearly 6,000 companies across every major industry. We partner with organizations to advise, deploy and operate complete cybersecurity programs from strategy and managed security services to risk, integration and technology solutions. With clients at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can. At Optiv, we manage cyber risk so you can secure your full potential. For more information, visit www.optiv.com. Related Insights Image Mastering the Hunt: Threat Hunting with Optiv and Carbon Black Even though corporate endpoints made up the top five assets involved in breaches last year, many enterprises still focus only on securing their network. Image Cyber Incident Readiness Services Cybersecurity incident readiness services measure your incident response capability against the current threat landscape and industry best practices. Learn more! Image Threat Intelligence Services Cybersecurity success hinges on transforming data into actionable intel. Gain resources to repel advanced cyber threats with Optiv’s Threat Intelligence services.
Optiv Security: Secure greatness.® Optiv is the cyber advisory and solutions leader, delivering strategic and technical expertise to nearly 6,000 companies across every major industry. We partner with organizations to advise, deploy and operate complete cybersecurity programs from strategy and managed security services to risk, integration and technology solutions. With clients at the center of our unmatched ecosystem of people, products, partners and programs, we accelerate business progress like no other company can. At Optiv, we manage cyber risk so you can secure your full potential. For more information, visit www.optiv.com.
Would you like to speak to an advisor? Let's Talk Cybersecurity Provide your contact information and we will follow-up shortly. Let's Browse Cybersecurity Just looking? Explore how Optiv serves its ~6,000 clients. Show me AI Security Solutions Show me the Optiv brochure Take me to Optiv's Events page Browse all Services