Automation on Willis Vandevanter

Pulling Specific Files from the Trickest Inventory (or any Github project)

Fri, 26 Aug 2022 00:00:00 +0000

The @trickest Inventory project is an interesting resource. It has a massive set of hostnames, live services, spidered URLs, and cloud data organized by Bug Bounty program. There is so much more data than I have interest in storing for my needs. In fact, the only thing I am interested in is the hostnames resource. Here is a quick and dirty way to pull the hostnames.txt file from every program without cloning the entire project.

💡 There is a good chance I am going to embarrass myself with this post and there is a better way. But this is part of learning and I embrace it. Please let me know and I will post the faster way at the top.

First, pull the current project git history without cloning it:

1
2
3
4


git clone --no-checkout \
--depth 1 \
--single-branch --branch=main \
https://github.com/trickest/inventory.git

Above we are cloning the project without checking out any files; --no-checkout. We are also only pulling HEAD (--depth 1) and only focused on the main branch.

Note, just the commit history from main takes up 336Mb 😲

Finally, we are going to download every hostname.txt file. This is done by finding the listing the HEAD tree, grep’ing for the filename, urlencoding & , and then downloading the file.

1
2
3
4


git ls-tree --full-name --name-only -r HEAD | \
grep hostnames.txt | \
sed -e "s/&/%26/" | \
xargs -I {} sh -c 'curl -o $(echo {} | cut -d\/ -f1)_hostnames.txt https://raw.githubusercontent.com/trickest/inventory/main/{}'

At this point you should have a directory full of the relevant files.

💡 If you are using this technique with another project take care that you trust the input (directories and filenames). You are piping them into a subshell.

If you don’t want to pipe into a subshell (yolo), you can use wget (remove the -o subshell) but you will be left with every file named hostnames.txt.X:

1
2
3
4


git ls-tree --full-name --name-only -r HEAD | \
grep hostnames.txt | \
sed -e "s/&/%26/" | \
xargs -I {} wget https://raw.githubusercontent.com/trickest/inventory/main/{}

🎉 burpsuite-project-file-parser v1.1 🎉

Thu, 21 Jul 2022 00:00:00 +0000

Edit: 1.1b fixes an auto shutdown issue in burpsuite, I would highly recommend this release over 1.1a. The rest of the post still applies.

This is a small release but a useful one.

Release 1.1b adds the ability to parse projects for portions of siteMap and proxyHistory. For example, the following will only respond with the proxyHistory request.headers and request.body. Note, the URL is always included:

1
2


java -jar -Djava.awt.headless=true [PATH_TO burpsuite_pro.jar] --project-file=[PATH TO PROJECT FILE] \
 proxyHistory.request.headers, proxyHistory.request.body

This should result in significant speed improvements as parsing will ignore response.body which can be very large. Conversely, if you only wanted to parse the proxyHistory response body for interesting things you could do:

1
2


java -jar -Djava.awt.headless=true [PATH_TO burpsuite_pro.jar] --project-file=[PATH TO PROJECT FILE] \
 proxyHistory.response.body

Building on an AppSec Pipeline with Burp Suite data - Part 2

Fri, 17 Jun 2022 00:00:00 +0000

In this two part series we are going to take Burp Suite Project files as input from the command line, parse them, and then feed them into a testing pipeline.

The series is broken down into two parts:

Getting at the Data (i.e. from the CLI to feeding the pipeline)
8 Bug Hunting Examples with burpsuite-project-parser (i.e. from the pipeline to testing)

This post is focused on bug hunting examples. Check out the previous post if you haven’t already setup the environment.

Command Shortcut

In the previous post we used a long (repetitive) command to print the auditItems from a Burp Suite project file:

1
2
3
4
5
6
7
8


java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file=2022-06-08.burp \
auditItems

For the sake of brevity, in this post we will replace the long command with a shorter one (e.g. $PARSE_BURP). You will need to make this specific to your environment:

1

export PARSE_BURP="java -jar -Djava.awt.headless=true -Xmx2G --add-opens=java.desktop/javax.swing=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED [INSERT_FULL_PATH]/burpsuite_pro_v2022.3.6.jar --user-config-file=[INSERT_FULL_PATH]/ONLY_BURP_PROJECT_PARSER.json --project-file=[INSERT_FULL_PATH]/[PROJECT_FILE].burp"

Then we can print all of the auditItems with:

1

$PARSE_BURP auditItems

8 Bug Hunting Examples with burpsuite-project-parser

⛅ This list does not try to be comprehensive. Smarter people than me have done much better work mind mapping bug hunting techniques. In fact, if anything these are incomplete. They are meant as starting points in taking input from a Burp Suite Project file to “looking for a bug or testing for a state” (i.e. pipeline). If your feeling is “I could do this better” you are probably right ha. Take what works for you and leave the rest 😊.

1. Base Case

In the base case burpsuite-project-parser proxyHistory will print the entire request (i.e. URL, headers, etc.) and response (headers, body, etc.) as JSON. For example:

1
2
3
4
5
6
7


$PARSE_BURP proxyHistory 2>/dev/null | grep -F "{" | head -n 2

...
...

{"Message":"Loaded project file parser; updated for burp 2022."}
{"request":{"url":"http://detectportal.firefox.com:80/success.txt","headers":["Host: detectportal.firefox.com","User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:80.0) Gecko/20100101 Firefox/80.0","Accept: */*","Accept-Language: en-US,en;q\u003d0.5","Accept-Encoding: gzip, deflate","Cache-Control: no-cache","Pragma: no-cache","Connection: close"],"uri":"/success.txt","method":"GET","httpVersion":"HTTP/1.1","body":""},"response":{"url":"http://detectportal.firefox.com:80/success.txt","headers":["Content-Type: text/plain","Content-Length: 8","Last-Modified: Mon, 15 May 2017 18:04:40 GMT","ETag: \"ae780585fe1444eb7d28906123\"","Accept-Ranges: bytes","Server: AmazonS3","X-Amz-Cf-Pop: ORD53-","X-Amz-Cf-Id: ADZK","Cache-Control: no-cache, no-store, must-revalidate","Date: Mon, 14 Sep 2020 17:59:54 GMT","Connection: close"],"code":"200","body":"success\n"}}

You will notice later on that we pipe this result to jq to get more specific with our query. For example, “give me only the URL from the JSON request” : | jq -c '{"url":.request.url}'). Although we could grep all of the requests and responses, the chances are we can be more surgical than that.

⚠️ I created an issue in burpsuite-project-parser to filter components from the proxyHistory and siteMap without jq. This should make the tool faster as well. You can follow the issue here and I will update the blog when this is done. ⚠️

2. Search for bug class specific GET Parameters

Like many people I have bug class specific GET parameters I search for (e.g. url= for SSRF). Let’s say we wanted to search a Burp Suite project for any request with url= as a GET parameter:

1
2
3
4


$PARSE_BURP proxyHistory 2>/dev/null \
| grep -F "{" | jq -c '{"url":.request.url}' \
| cut -d\" -f4 | tr -d \" \
| grep -ie "\?url=" -ie "\&url=" 

Example Results:

1
2
3


https://target1:443/pagead/1p-user-list/1057924016/?url=example
https://target1:443/cc.js?engine_key=123Q2K&url=somesite
...

Let’s break this first example down a bit.

$PARSE_BURP proxyHistory 2>/dev/null–> Parse our project file and output all of the request/response Proxy History as JSON
| grep -F "{" | jq -c '{"url":.request.url}'–> Take the JSON input and grab** only the request URLs**
| cut -d" -f4 | tr -d \"–> Give me the URL only and trim the quotes
grep -ie "?url=" -ie "&url="–> Grep for either (-e) “?url=” or “&url” in a case insensitive manner

This should give us a nice list of URLs that contained =url in their GET request.

💡 You can replace the above grep command with any bug class you find interesting. Resources like SecLists are a good start with example dictionaries. There are a lot more out there as well and I think most people curate their own.

3. Create a script to request a page with input from proxy history

Let’s say we wanted to take in every URL from our project and perform a scan looking for a specific file (e.g. /.git/config) on that URL. Here is one way to create a script for this using the previous Burp History as input in our pipeline.

1
2
3
4
5
6


$PARSE_BURP proxyHistory 2>/dev/null \
| grep -F "{" | jq -c '{"url":.request.url}' \
| cut -d\" -f4 | tr -d \" \
| cut -d\? -f1 \ 
| xargs -I {} printf "curl {}/.git/config\n"
| tee git_script.sh

You should end up with set of commands in a shell script like:

1
2
3
4


curl https://target1.com/images/font-awesome-4.2.0/fonts/fontawesome-webfont.woff/.git/config
curl https://target1.com/images/avatar.png/.git/config
curl https://target1.com/some/dir/.git/config
...

Right away you can probably see one of the (many) problems with this. Our “pipeline” is appending to the full URL and not cutting off at the directory. In some cases this might be intended behavior, but chances are it is not.

💡 I will leave it as an exercise to the reader to fix this (hint: rev + cut complement is one way. The solution is also in the next section).What other problems could there be with doing it this way? Are these the best settings for curl? Is curl the best tool for this job?

4. Feeding the ffuf monster

ffuf is incredible. Read/watch this brilliant 💎 by @codingo for an overview of ffuf: https://codingo.io/tools/ffuf/bounty/2020/09/17/everything-you-need-to-know-about-ffuf.html.

The previous idea of searching for a specific file is better suited for a tool like ffuf. So let’s go back to the same page search but with ffuf instead. First, make sure to create a “bruteforce dictionary” with the just /.git/config in it. Then:

1
2
3
4
5
6
7


$PARSE_BURP proxyHistory 2>/dev/null \
| grep -F "{" | jq -c '{"url":.request.url}' \
| cut -d\" -f4 | tr -d \" \
| rev | cut -d\/ -f2- | rev \
| sort -u --parallel=2G \
| xargs -I {} printf "ffuf -t 40 -r -u \"{}/FUZZ\" -maxtime 60 -v -c -w /tmp/gitc \n" \
| tee ffuf_search.sh

Example results:

1
2
3
4


ffuf -t 40 -r -u "http://target1/ajax/libs/jquery/1.11.0/FUZZ" -maxtime 60 -v -c -w /tmp/gitc
ffuf -t 40 -r -u "http://target2/FUZZ" -maxtime 60 -v -c -w /tmp/gitc
ffuf -t 40 -r -u "http://target2/images/FUZZ" -maxtime 60 -v -c -w /tmp/gitc
...

| rev | cut -d\/ -f2- | rev \–> This is the solution to the previous question; grab the URL up to the directory
| sort -u --parallel=2G–> Sort and give only the unique URLs.
xargs -I {} printf "ffuf -t 40 -r -u "{}/FUZZ" -maxtime 60 -v -c -w /tmp/gitc \n"–> The ffuf command

💡 What are my assumptions and potential issues with this new technique? How is this inefficient? Is every URL in-scope to your testing? Is the ffuf command correct?

5. Find HTTP Response Headers with nginx

In this example we want to look through a Burp Suite project for any server response header that contains nginx (i.e. Server: Nginx 1.12 ). This can be done with:

1
2


$PARSE_BURP responseHeader='.*(Servlet|nginx).*' 2>/dev/null \
| sort -u --parallel=2G

Example Results:

1
2
3


{"url":"https://target1:443/webfonts/fa-solid-900.woff2","header":"Server: nginx/1.12.2"}
{"url":"https://target2:443/","header":"Server: nginx/1.14.0 + Phusion Passenger 6.0.6"}
...

6. Search for an API key with regex - Take 1

In this example we want to search through a Burp Suite Project for a known API key regex. For example, ([^A-Z0-9]|^)(AKIA|A3T|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{12,} (Source: https://github.com/dxa4481/truffleHogRegexes/issues/19) will identify AWS API keys. Here is how we would do that against our project file:

1

$PARSE_BURP responseHeader='.*([^A-Z0-9]|^)(AKIA|A3T|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{12,}.*' 2>/dev/null

7. Search for all the API key(s) with regex - Take 2

There are a couple of issues with Take 1 above. First, it has low yield because we are only using a single regex when we could be greedier about it. Second, it’s memory intensive and doesn’t scale well.

One solution is to use “the save results to MongoDB” feature (i.e. storeData=[MongoDB Host]) and then write a script to search the results. This scales very well and is reusable.

Another solution which is a little messier (and greedier) is to write all of the responses to files and then use an awesome tool like trufflehog (https://github.com/trufflesecurity/trufflehog) to find all the secrets. That sounds like more fun, let’s go with that.

Step 1 is to write all of the HTTP responses from a Burp Suite project file to a directory.

1
2
3
4


mkdir burp_responses
$PARSE_BURP proxyHistory 2>/dev/null \
| grep -F "{" | jq -c '{"url":.request.url,"body":.response.body}' \
| while read line; do echo $line | tee burp_responses/$(uuidgen | tr -d '-').burp; done

💡 Note, this will print the URL and the response body (only) to a set of files with one request/response per file. If you want to search HTTP request headers, HTTP response headers etc. then you need to adjust or remove the jq filter on line 2.On my system this command took around 10 minutes to run. A 384Mb project file became 313Mb worth of 105,309 files.⚠️ This is the first time I have broken ls on my system with a too many files error in a directory 😂 ⚠️

At this point we should have a directory (i.e. burp_responses) filled with thousands of files containing the URL and the response one per file. Lastly run trufflehog over the set of files and look for results.

1

trufflehog filesystem --directory=burp_responses --no-verification | tee trufflehog_results.txt

💡 For speed and privacy reasons, I chose to set the –no-verification flag on my first pass. On secondary passes I would likely remove this flag.

8. Search for all the API key(s) with regex - Take 3

Because we already have the HTTP response bodies in files let’s use gf by the legend @tomnomnom to search for interesting things. If you are unfamiliar with gf, the core idea is it’s a reusable wrapper around grep.

gf comes pre-packaged with a set of great checks; https://github.com/tomnomnom/gf/blob/master/examples/.

Let’s run the s3-buckets common gf patterns over our HTTP responses and see if we find anything of interest:

1
2
3
4


cd burp_responses
gf s3-buckets \
| sort -u --parallel=2G \
| tee -a gf_results.txt

Although it’s not as powerful as using trufflehog, it is far superior to take 1. Furthermore, gf makes it easy to write and reuse your own grep checks. Consider this option when reviewing for interesting things at scale.

Concluding Thoughts

We have just skimmed the surface of the automation capabilities. I have lot more ideas (and experience) related to AppSec automation, so stay tuned!

Building on an AppSec Pipeline with Burp Suite data - Part 1

Wed, 08 Jun 2022 00:00:00 +0000

In this two part series we are going to take Burp Suite Project files as input from the command line, parse them, and then feed them into a testing pipeline.

The series is broken down into two parts:

Getting at the Data (i.e. from the CLI to feeding the pipeline)
8 Bug Hunting Examples with burpsuite-project-parser (i.e. from the pipeline to testing)

Introduction

Two years ago I pushed to Github a Burp Suite plugin with a mouthful of a name: burpsuite-project-parser. It started out to solve a very simple problem.

I am on day 10 of a web application assessment, I intercept a request, and I ask myself “Where the $@#* have I seen that parameter before?!?!”

When I am on a long assessment or bug hunting over a period of time I keep multiple sequential Burp project files (e.g. 06-01-2022.burp, 06-08-2022.burp, etc). Typically I would need to open and close Burp Suite for each project file using the search UI to hunt for this single parameter or URI. This led to the idea:

💡 What if you could output as JSON all of the requests, responses, and findings from a Burp Suite project file using the CLI and then grep to search? Or save to a database? Or feed to another tool? ….

This was the first problem solved by burpsuite-project-parser. From the CLI it will output every request/response (or findings) from a project file. For example:

1
2
3
4


...
{"request":{"url":"http://secret.targethost.com:80/success.txt","headers":["Host: secret.targethost.com","User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:80.0) Gecko/20100101 Firefox/80.0","Accept: */*","Accept-Language: en-US,en;q\u003d0.5","Accept-Encoding: gzip, deflate","Cache-Control: no-cache","Pragma: no-cache","Connection: close"],"uri":"/success.txt","method":"GET","httpVersion":"HTTP/1.1","body":""},"response":{"url":"http://secret.targethost.com:80/success.txt","headers":["Content-Type: text/plain","Content-Length: 8","Last-Modified: Mon, 15 May 2017 18:04:40 GMT","ETag: \"ae780585fe1444eb7d28906123\"","Accept-Ranges: bytes","Server: AmazonS3","X-Amz-Cf-Pop: ORD53-","X-Amz-Cf-Id: ADZK","Cache-Control: no-cache, no-store, must-revalidate","Date: Mon, 14 Sep 2020 17:59:54 GMT","Connection: close"],"code":"200","body":"success\n"}}
{"request":{"url":"https://mail.targethost.com:443/somepage","headers":["Host: x.tesla.com:443/somepage","User-Agent: ...
...

The Github page for burpsuite-project-parser has the most up to date installation instructions so I won’t repeat those here. Instead I want to talk about how to parse larger amounts of Burp data in our pipeline.

Moving Faster with Burp Suite User-Level Configuration

Photo by Anton Filatov / Unsplash

You may or may not have played with Burp User-Level Configurations; they were certainly new to me when I started this project. The Burp Suite documentation does the best job of describing what’s included so I will just screenshot it here:

BurpSuite Documentation Screenshot taken on 06/05/22

The most important point is that we can create a User-Level configuration to include just the burpsuite-project-parser Extender tool and not break our default Burp Suite configuration. This allows the loading and unloading of Burp Suite to be much faster as we are only applying one extension against the project file.

The following assumes you have already installed Burp Suite Project File Parser; if not, install it before going forward.

First, **save your existing user options; **Burp > User Options > Save user options. Use a memorable name such as “DEFAULT_BURP_USER_OPTIONS.json”:

My default setup

Next, disable all other Extensions except “BurpSuite Project File Parser” and “Save user options” as a new file (i.e. “ONLY_BURP_PROJECT_PARSER.json”):

Only the Project File Parser tool

Finally, before closing Burp Suite, click “Load user options” and load your original custom options (i.e. “DEFAULT_BURP_USER_OPTIONS.json”) back in. This way, the next time you open Burp Suite GUI your configuration will be the same as what you are used to.

Testing

It’s time to test our setup. Run the following command against an existing project file to verify everything is working correctly. Make sure to replace 2022-06-08.burp with the name of your Burp Suite Project file and the location of your burpsuite jar file (e.g. ~/Downloads/burpsuite_pro_v2022.3.6.jar below):

1
2
3
4
5
6
7
8


java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file=2022-06-08.burp \
auditItems

You should see audit items in the result:

1
2
3
4
5
6


Warning: the fonts "Times" and "Times" are not available for the Java logical font "Serif", which may have unexpected appearance or behavior. Re-enable the "Times" font to remove this warning.
{"Message":"Loaded project file parser; updated for burp 2022."}
[auditItems]
{"issueName":"Unencrypted communications","url":"http://site1:80/","confidence":"Certain","severity":"Low"}
{"issueName":"Unencrypted communications","url":"http://site2:80/","confidence":"Certain","severity":"Low"}
...

💡 Using my laptop as a benchmark it took around half the time to process a project file using the single purpose user option configuration compared to default. This speed up is even more drastic when we begin to process more files, larger projects, and include more complex options.

Maybe not light speed but it’s waayyy faster.

burpsuite-project-parser Flags

At this point we now have a speedier way to parse project files. Before giving a few examples let’s reiterate what flags are available as of burpsuite-project-parser 1.0. Remember any output will be in JSON:

auditItems: Outputs the audit findings from a project file. siteMap: Outputs all requests/responses from the site map. proxyHistory: Outputs all requests/responses from the site map. responseHeader=[regex]: Using the [regex] output any response that matches in the response headers. responseBody=[regex]: Using the [regex] output any response that matches in the response body. storeData=[MongoDB Host]: Store all requests/responses to a MongoDB server; check out the Github project for the required MongoDB settings.

Feeding Data into the Pipeline

Before we finish up let’s do a few examples.

Here is a bash one-liner to output all of the findings from all of the project files in the current directory:

Linux:

1
2
3
4
5
6
7
8
9


find . -maxdepth 1 -name "*.burp" | xargs -i{} \ 
java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file={} \
auditItems 2>/dev/null

OS X:

1
2
3
4
5
6
7
8
9


find . -maxdepth 1 -name "*.burp" | xargs -I{} \
java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file={} \
auditItems 2>/dev/null

Search every project file for Servlet or nginx in response header:

1
2
3
4
5
6
7
8
9


find . -maxdepth 1 -name "*.burp" | xargs -I{} \
java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file={} \
responseHeader='.*(Servlet|nginx).*' 2>/dev/null

Grep all proxyHistory of all project files for “graphql” anywhere and output the URL where it was seen:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


find . -maxdepth 1 -name "*.burp" | xargs -I{} \
java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file={} \
proxyHistory 2>/dev/null \
| grep -Fi "graphql" \
| jq -c '{"url":.request.url}' | cut -d\" -f4

Grep for “url=” from proxyHistory in the url and uri only:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


find . -maxdepth 1 -name "*.burp" | xargs -I{} \
java -jar -Djava.awt.headless=true \
-Xmx2G \
--add-opens=java.desktop/javax.swing=ALL-UNNAMED \
--add-opens=java.base/java.lang=ALL-UNNAMED \
~/Downloads/burpsuite_pro_v2022.3.6.jar \
--user-config-file=ONLY_BURP_PROJECT_PARSER.json \
--project-file={} \
proxyHistory 2>/dev/null \
| grep -F "{" \ 
| jq -c '{"url":.request.url,"uri":.request.uri}' \
| cut -d\" -f4,8 | tr -d \" \
| grep -iF "url="

YMMV

Please keep in mind this plug-in follows a design philosophy of “one tool for the job”. Grepping through the proxyHistory and only outputting a URL may not be the most accurate way to get the data you are looking for. Instead, maybe putting everything into MongoDB (ElasticSearch, etc) or a custom JSON search script works better. In this next post we will take this idea further.

Please submit bugs and improvements to the Github project if you want!

odle ruby gem: piping security data

Thu, 24 May 2018 00:00:00 +0000

I recently (May 2018) published odle which is a Ruby gem and binary that takes XML data from various security tools and outputs their JSON equivalent. The goal is to be (1) simple, (2) fast, and (3) work on many platforms with only one dependency – nokogiri.

Quick Example of Piping Security Results

Below are two examples using odle to convert output from one tool (e.g. burpsuite) as input for something else (e.g. nmap scans). From the command line I typically use odle with gron which is an awesome tool that “makes json greppable” =).

Convert Burp to nmap script scan

Often I will take the passive data from one tool and feed it into another tool. One example is burp to something else; in this case, nmap script checks.

1
2
3
4
5
6
7
8


cat burp-scan.xml | odle --burp | gron | grep -i 'affected_hosts' | cut -d \" -f4 | cut -d/ -f3 | cut -d' ' -f1 | sort | uniq | xargs printf "nmap -sS -Pn -p 21 --script +ftp-anon %s \n"

nmap -sS -Pn -p 21 --script +ftp-anon apis.google.com 
nmap -sS -Pn -p 21 --script +ftp-anon developer.cdn.mozilla.net 
nmap -sS -Pn -p 21 --script +ftp-anon fakesite.com 
nmap -sS -Pn -p 21 --script +ftp-anon fonts.googleapis.com 
nmap -sS -Pn -p 21 --script +ftp-anon safebrowsing-cache.google.com 
nmap -sS -Pn -p 21 --script +ftp-anon safebrowsing.google.com 

Run nessus results through aquatone

1
2
3
4
5


cat nessus_v2.xml | odle --nessus | ~/Downloads/gron | grep -i 'affected_hosts' | cut -d \" -f4 | cut -d/ -f3 | cut -d' ' -f1 | sort | uniq | xargs printf "aquatone --discover %s \n"

aquatone --discover admin.fb.com
aquatone --discover js.fb.com
aquatone --discover blah.fb.com

Install

1
2


gem install nokogiri
gem install odle

Bugs

I am sure there are plenty. Please submit an issue if you find one or if you would like to see other supported tools. I am also interested in inconsistencies between outputs, missing data, and other issues if you see them.

Exploiting CVE-2016-4264 With OXML_XXE

Sun, 02 Oct 2016 00:00:00 +0000

Recently ColdFusion was shown vulnerable to XXE based attacks in OXML documents; CVE-2016-4264. The blog post linked gives an example building the file using python; cool!

It’s easy to backdoor files in a similar fashion with OXML XXE. The fastest way to do this is using the “Overwrite File inside DOCX/etc” function.

You can add any XLSX at this point, OXML_XXE ships with a sample.xlsx.

You will want to specify the XML file to overwrite; e.g. “[Content_Types.xml]”. The “_rels/.rels” file is another option.

Finally add in the XML exploit. Below a Parameter Entity is used.

Click “Build” to generate and download the file. To verify that the file is sound you can view the generated file in the “List Previously Built Files” menu option.

Finding Hosts Using SSL Certificate Organization And Censys

Tue, 27 Sep 2016 00:00:00 +0000

Finding hosts or domain names associated with a company where the domain name does not include the name of the company can sometimes be difficult. There are common ways to do it such as ASN or scope information (e.g. bug bounty ToE or IP block).

One technique that I use (and I am guessing others do as well) is through an Organization field in a SSL Certificate that is shared by multiple domains. For example, a certificate from https://www.facebook.com and https://parse.com are signed by the same organization.

Notice the Organization in the SSL Certificate and compare to the image below

Notice the Organization in the SSL Certificate and compare to the image above

This is an easy example. Parse is listed on the Facebook Mergers or Acquisitions page (https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitions_by_Facebook) and the FB bug bounty terms (https://www.facebook.com/whitehat).

However, consider a more complex example like “HackerOne: Subdomain takeover on https://fastly.sc-cdn.net/”.

First, if you aren’t familiar with sub-domain take over this and this are awesome.

In the case above, the subdomain fastly.sc-cdn.net is owned by Snapchat which is not obvious from the domain name. Personally, I do not know how ebrietas found that domain. DNS Bruteforcing would work. It could also be done by shared SSL Certificates on the Organization name.

A few months back I wrote a script that uses the Censys API to look for domains with the same Organization field in the SSL certificate.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


ruby censys_cert_search.rb 'Snapchat Inc.'

Page: 1
{"parsed.extensions.subject_alt_name.dns_names"=>["se.snap-dev.net"], "parsed.subject.common_name"=>["se.snap-dev.net"]}
|+| Checking se.snap-dev.net
se.snap-dev.net. 300 IN A 107.178.248.183
se.snap-dev.net. 300 IN A 107.178.248.183
|+| Checking se.snap-dev.net
se.snap-dev.net. 300 IN A 107.178.248.183
se.snap-dev.net. 300 IN A 107.178.248.183
{"parsed.extensions.subject_alt_name.dns_names"=>["spectre.sc-corp.net"], "parsed.subject.common_name"=>["spectre.sc-corp.net"]}
|+| Checking spectre.sc-corp.net
spectre.sc-corp.net. 300 IN A 130.211.14.254
spectre.sc-corp.net. 300 IN A 130.211.14.254
|+| Checking spectre.sc-corp.net
spectre.sc-corp.net. 300 IN A 130.211.14.254
spectre.sc-corp.net. 300 IN A 130.211.14.254
...

If you checkout the code, the script is:

Using the Censys Certificate API to search on the Organization string matching ‘Snapchat Inc.’ (i.e. O=Snapchat Inc.*)
Parsing out the Common Name and Alternate Names from the SSL Certificate response
Performing a DNS lookup for each name found

You can also run the script to skip over common names that could’ve been easily found in other ways (e.g. dev.snapchat.com). This focuses the effort on hard to find systems. Adding a third argument skips DNS lookup.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


ruby censys_cert_search.rb 'Snapchat Inc.' '*snapchat.com' false

Page: 1
{"parsed.extensions.subject_alt_name.dns_names"=>["se.snap-dev.net"], "parsed.subject.common_name"=>["se.snap-dev.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["spectre.sc-corp.net"], "parsed.subject.common_name"=>["spectre.sc-corp.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.targeting.snapads.com", "targeting.snapads.com"], "parsed.subject.common_name"=>["*.targeting.snapads.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["rest-escluster.hydrasearch.sc-prod.net"], "parsed.subject.common_name"=>["rest-escluster.hydrasearch.sc-prod.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["restfulgit.sc-corp.net"], "parsed.subject.common_name"=>["restfulgit.sc-corp.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["attribution.snapads.com"], "parsed.subject.common_name"=>["attribution.snapads.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com", "mail.support.snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
....

As you can see not perfect but I’ve found it really useful in the past. It was hack for something else and I’ll update as time permits.

Cloud Metadata URL List

Mon, 28 Mar 2016 00:00:00 +0000

I landed the SSRF Cloud Metadata technique in a few different scenarios recently. If you haven’t seen the talk BHUSA 2014 - Bringing a Machete to the Amazon I recommend it.

To make life a little easier created a living URL list for Metadata broken down by cloud. There are a few more than he discusses in the talk but still has work to go. Submit a PR if you see some missing.

https://gist.github.com/BuffaloWill/fa96693af67e3a3dd3fb

Search all Github Repositories for an Organization

Fri, 09 Jan 2015 00:00:00 +0000

gumbler is a script I wrote to search through git commits and introduced in the blog post “Searching Through Git Commits”. Recently I wanted to run Gumbler across all repositories for an organization, the steps are discussed below.

First, we need to grab a list of repositories for the ORG. This can be done using the API

1
2
3


curl "https://api.github.com/orgs/[ORG NAME]/repos?page=1&per_page=10000" > repos.json
curl "https://api.github.com/orgs/[ORG NAME]/repos?page=2&per_page=10000" >> repos.json
...

Note, the API limits the number of values returned so you will want to update the page count to make sure you get them all.

Next we iterate through each repository, clone it, and run gumbler across the repo. A simple Ruby script is given below. Note, ignore any repos that were forked as they aren’t specific to the orgnization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


require 'json'

# specify repos.json as an argument
file = File.read(ARGV[0])

# parse the json
data_hash = JSON.parse(file)

# iterate each hash
data_hash.each do |hash|
 # ignore forked repos
 if !(hash["fork"])
 puts "|+| Testing #{hash["name"]}"
 
 # clone the project
 `git clone https://github.com/[ORG]/#{hash["name"]}.git`
 
 # Gumbler requires full directory paths
 `ruby ~/gumbler/gumbler.rb -s -p #{hash["name"]} ~/[ORG]/results/`

 sleep(3)
 end
end

Searching Through Git Commits

Mon, 06 Oct 2014 00:00:00 +0000

gumbler is a script I wrote to search through git commits. Examples from github are discussed below.

.gitignore

A gitignore file is used to specify files that should not be tracked by git (source gitignore). In the default case, gumbler will read the gitignore file for the project and search every revision for a case where a file from gitignore was committed. Possible use cases would be as a pen tester looking for reconnaisance data (e.g. developer usernames/passwords, staging hosts/services, etc.) or a developer to verify projects did not previously commit “secret” data.

I am a big fan of what Netflix is doing with regards to open source and security. After looking through a number of their projects, I noticied Priam has a few commits with non-damaging files from the gitignore.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


$. git clone https://github.com/Netflix/Priam.git
Cloning into 'Priam'...
....
Checking connectivity... done.
$. ruby gumbler/gumbler.rb Priam/ gumbler_testing/tmp/
|-| Jumping to remote @directory Priam/
|-| Storing every revision
checking for *.com..
checking for *.class..
.....
|+| Looking for .classpath, Found it in BRANCH : 697fd66aae9beed107e13f49a741455f1d9d8dd9 .classpath. Storing it in gumbler_testing/tmp/.
|+| Looking for .classpath, Found it in BRANCH : 47bdb537789c034493e94d8977eae77ecbfd5b24 .classpath. Storing it in gumbler_testing/tmp/.
|+| Looking for .classpath, Found it in BRANCH : 442862d4a8d4d18d0e176ded8795dd45a24528fc .classpath. Storing it in gumbler_testing/tmp/.
....
checking for .project..
|+| Looking for .project, Found it in BRANCH : 697fd66aae9beed107e13f49a741455f1d9d8dd9 .project. Storing it in gumbler_testing/tmp/.
|+| Looking for .project, Found it in BRANCH : 0941d9e0e0dda3ee1d9d4dda757d59ffb641abcf .project. Storing it in gumbler_testing/tmp/.
|+| Looking for .project, Found it in BRANCH : 47bdb537789c034493e94d8977eae77ecbfd5b24 .project. Storing it in gumbler_testing/tmp/.
....
checking for .settings..
....

.classpath or .project are not damaging in this case and, hence, are used as the example. On a pen test or in collaborative projects I have found much worse (cough usernames, passwords). This shouldn’t be that surprising.

Searching Commit Logs

Another use case for gumbler is to look through commit history. Using Ruby on Rails as an example, we can search from for any commit with “CVE” in it. Gumbler will output a diff from the files changed in the commit.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


$. ruby gumbler/gumbler.rb --grep CVE rails/ tmp/
|!| skipping .gitignore, searching commit log for CVE
$. ls tmp/
060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30_15:13:03_-0700.diff 88cc1688d0cb828c17706b41a8bd27870f2a2beb-2013-01-08_12:11:18_-0800.diff
08d0a11a3f62718d601d39e617c834759cf59bbb-2014-02-18_15:38:50_-0300.diff 8be6913990c30f63618173da722148892348dcc9-2013-03-15_17:45:53_-0700.diff
0b58a7ff420d7ef4b643c521a62be7259dd2f5cb-2011-02-08_14:21:12_-0800.diff 8e577fe560d5756fcc67840ba304d79ada6804e4-2013-01-08_12:41:24_-0800.diff
0c7ac34aed1845044cd1911e5a775366d7ca41c1-2013-12-02_16:42:16_-0800.diff 9340f89849606dba02f44038171f3837f883fd4e-2012-05-30_15:09:13_-0700.diff
2392535f4085d88186097e3c23414e958fb1d16d-2013-03-18_10:17:32_-0700.diff 93fb4c1e62dc9605eecbfaffda2becc85890fa5f-2014-07-10_10:20:16_-0700.diff
...
$. cat 060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30_15\:13\:03_-0700.diff 
060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30 15:13:03 -0700==> 2012-05-30 15:13:03 -0700 Strip [nil] from parameters hash. Thanks to Ben Murphy for reporting this! Strip [nil] from parameters hash.
Thanks to Ben Murphy for reporting this!

CVE-2012-2660

:100644 100644 aa5ba3e... 6757a53... M actionpack/lib/action_dispatch/http/request.rb
:100644 100644 c3f009a... 6ea66f9... M actionpack/test/dispatch/request/query_string_parsing_test.rb

diff --git a/actionpack/lib/action_dispatch/http/request.rb b/actionpack/lib/action_dispatch/http/request.rb
index aa5ba3e..6757a53 100644
--- a/actionpack/lib/action_dispatch/http/request.rb
+++ b/actionpack/lib/action_dispatch/http/request.rb
....

As the README says, be careful using the tool as it uses Command Execution to search. A malicious git project could take advantage of this. Ping me with better ways to handle this.