Recon on Willis Vandevanter

Pulling Specific Files from the Trickest Inventory (or any Github project)

Fri, 26 Aug 2022 00:00:00 +0000

The @trickest Inventory project is an interesting resource. It has a massive set of hostnames, live services, spidered URLs, and cloud data organized by Bug Bounty program. There is so much more data than I have interest in storing for my needs. In fact, the only thing I am interested in is the hostnames resource. Here is a quick and dirty way to pull the hostnames.txt file from every program without cloning the entire project.

💡 There is a good chance I am going to embarrass myself with this post and there is a better way. But this is part of learning and I embrace it. Please let me know and I will post the faster way at the top.

First, pull the current project git history without cloning it:

1
2
3
4


git clone --no-checkout \
--depth 1 \
--single-branch --branch=main \
https://github.com/trickest/inventory.git

Above we are cloning the project without checking out any files; --no-checkout. We are also only pulling HEAD (--depth 1) and only focused on the main branch.

Note, just the commit history from main takes up 336Mb 😲

Finally, we are going to download every hostname.txt file. This is done by finding the listing the HEAD tree, grep’ing for the filename, urlencoding & , and then downloading the file.

1
2
3
4


git ls-tree --full-name --name-only -r HEAD | \
grep hostnames.txt | \
sed -e "s/&/%26/" | \
xargs -I {} sh -c 'curl -o $(echo {} | cut -d\/ -f1)_hostnames.txt https://raw.githubusercontent.com/trickest/inventory/main/{}'

At this point you should have a directory full of the relevant files.

💡 If you are using this technique with another project take care that you trust the input (directories and filenames). You are piping them into a subshell.

If you don’t want to pipe into a subshell (yolo), you can use wget (remove the -o subshell) but you will be left with every file named hostnames.txt.X:

1
2
3
4


git ls-tree --full-name --name-only -r HEAD | \
grep hostnames.txt | \
sed -e "s/&/%26/" | \
xargs -I {} wget https://raw.githubusercontent.com/trickest/inventory/main/{}

Finding Hosts Using SSL Certificate Organization And Censys

Tue, 27 Sep 2016 00:00:00 +0000

Finding hosts or domain names associated with a company where the domain name does not include the name of the company can sometimes be difficult. There are common ways to do it such as ASN or scope information (e.g. bug bounty ToE or IP block).

One technique that I use (and I am guessing others do as well) is through an Organization field in a SSL Certificate that is shared by multiple domains. For example, a certificate from https://www.facebook.com and https://parse.com are signed by the same organization.

Notice the Organization in the SSL Certificate and compare to the image below

Notice the Organization in the SSL Certificate and compare to the image above

This is an easy example. Parse is listed on the Facebook Mergers or Acquisitions page (https://en.wikipedia.org/wiki/List_of_mergers_and_acquisitions_by_Facebook) and the FB bug bounty terms (https://www.facebook.com/whitehat).

However, consider a more complex example like “HackerOne: Subdomain takeover on https://fastly.sc-cdn.net/”.

First, if you aren’t familiar with sub-domain take over this and this are awesome.

In the case above, the subdomain fastly.sc-cdn.net is owned by Snapchat which is not obvious from the domain name. Personally, I do not know how ebrietas found that domain. DNS Bruteforcing would work. It could also be done by shared SSL Certificates on the Organization name.

A few months back I wrote a script that uses the Censys API to look for domains with the same Organization field in the SSL certificate.

For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


ruby censys_cert_search.rb 'Snapchat Inc.'

Page: 1
{"parsed.extensions.subject_alt_name.dns_names"=>["se.snap-dev.net"], "parsed.subject.common_name"=>["se.snap-dev.net"]}
|+| Checking se.snap-dev.net
se.snap-dev.net. 300 IN A 107.178.248.183
se.snap-dev.net. 300 IN A 107.178.248.183
|+| Checking se.snap-dev.net
se.snap-dev.net. 300 IN A 107.178.248.183
se.snap-dev.net. 300 IN A 107.178.248.183
{"parsed.extensions.subject_alt_name.dns_names"=>["spectre.sc-corp.net"], "parsed.subject.common_name"=>["spectre.sc-corp.net"]}
|+| Checking spectre.sc-corp.net
spectre.sc-corp.net. 300 IN A 130.211.14.254
spectre.sc-corp.net. 300 IN A 130.211.14.254
|+| Checking spectre.sc-corp.net
spectre.sc-corp.net. 300 IN A 130.211.14.254
spectre.sc-corp.net. 300 IN A 130.211.14.254
...

If you checkout the code, the script is:

Using the Censys Certificate API to search on the Organization string matching ‘Snapchat Inc.’ (i.e. O=Snapchat Inc.*)
Parsing out the Common Name and Alternate Names from the SSL Certificate response
Performing a DNS lookup for each name found

You can also run the script to skip over common names that could’ve been easily found in other ways (e.g. dev.snapchat.com). This focuses the effort on hard to find systems. Adding a third argument skips DNS lookup.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


ruby censys_cert_search.rb 'Snapchat Inc.' '*snapchat.com' false

Page: 1
{"parsed.extensions.subject_alt_name.dns_names"=>["se.snap-dev.net"], "parsed.subject.common_name"=>["se.snap-dev.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["spectre.sc-corp.net"], "parsed.subject.common_name"=>["spectre.sc-corp.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.targeting.snapads.com", "targeting.snapads.com"], "parsed.subject.common_name"=>["*.targeting.snapads.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["rest-escluster.hydrasearch.sc-prod.net"], "parsed.subject.common_name"=>["rest-escluster.hydrasearch.sc-prod.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["restfulgit.sc-corp.net"], "parsed.subject.common_name"=>["restfulgit.sc-corp.net"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["attribution.snapads.com"], "parsed.subject.common_name"=>["attribution.snapads.com"]}
{"parsed.extensions.subject_alt_name.dns_names"=>["*.snapchat.com", "snapchat.com", "mail.support.snapchat.com"], "parsed.subject.common_name"=>["*.snapchat.com"]}
....

As you can see not perfect but I’ve found it really useful in the past. It was hack for something else and I’ll update as time permits.

ldapsearch notes

Wed, 25 Feb 2015 00:00:00 +0000

I seem to find open LDAP servers on the Internet more often than I should. Here are some notes on using ldapsearch

Installing ldapsearch on Ubuntu

1

apt-get install ldap-utils

Root-DSE object

nmap includes a script to gather info from a LDAP root-dse object (http://nmap.org/nsedoc/scripts/ldap-rootdse.html). We can also use ldapsearch to test:

1

ldapsearch -p [PORT] -x -b "" -s base 'objectclass=*' -h [IP]

Open LDAP server

Connect to an open LDAP server, john the ripper can be used to crack passwords that are returned:

1

ldapsearch -p [PORT] -x -h [IP] -b "dc=[y],dc=com"

Search all Github Repositories for an Organization

Fri, 09 Jan 2015 00:00:00 +0000

gumbler is a script I wrote to search through git commits and introduced in the blog post “Searching Through Git Commits”. Recently I wanted to run Gumbler across all repositories for an organization, the steps are discussed below.

First, we need to grab a list of repositories for the ORG. This can be done using the API

1
2
3


curl "https://api.github.com/orgs/[ORG NAME]/repos?page=1&per_page=10000" > repos.json
curl "https://api.github.com/orgs/[ORG NAME]/repos?page=2&per_page=10000" >> repos.json
...

Note, the API limits the number of values returned so you will want to update the page count to make sure you get them all.

Next we iterate through each repository, clone it, and run gumbler across the repo. A simple Ruby script is given below. Note, ignore any repos that were forked as they aren’t specific to the orgnization.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


require 'json'

# specify repos.json as an argument
file = File.read(ARGV[0])

# parse the json
data_hash = JSON.parse(file)

# iterate each hash
data_hash.each do |hash|
 # ignore forked repos
 if !(hash["fork"])
 puts "|+| Testing #{hash["name"]}"
 
 # clone the project
 `git clone https://github.com/[ORG]/#{hash["name"]}.git`
 
 # Gumbler requires full directory paths
 `ruby ~/gumbler/gumbler.rb -s -p #{hash["name"]} ~/[ORG]/results/`

 sleep(3)
 end
end

Searching Through Git Commits

Mon, 06 Oct 2014 00:00:00 +0000

gumbler is a script I wrote to search through git commits. Examples from github are discussed below.

.gitignore

A gitignore file is used to specify files that should not be tracked by git (source gitignore). In the default case, gumbler will read the gitignore file for the project and search every revision for a case where a file from gitignore was committed. Possible use cases would be as a pen tester looking for reconnaisance data (e.g. developer usernames/passwords, staging hosts/services, etc.) or a developer to verify projects did not previously commit “secret” data.

I am a big fan of what Netflix is doing with regards to open source and security. After looking through a number of their projects, I noticied Priam has a few commits with non-damaging files from the gitignore.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


$. git clone https://github.com/Netflix/Priam.git
Cloning into 'Priam'...
....
Checking connectivity... done.
$. ruby gumbler/gumbler.rb Priam/ gumbler_testing/tmp/
|-| Jumping to remote @directory Priam/
|-| Storing every revision
checking for *.com..
checking for *.class..
.....
|+| Looking for .classpath, Found it in BRANCH : 697fd66aae9beed107e13f49a741455f1d9d8dd9 .classpath. Storing it in gumbler_testing/tmp/.
|+| Looking for .classpath, Found it in BRANCH : 47bdb537789c034493e94d8977eae77ecbfd5b24 .classpath. Storing it in gumbler_testing/tmp/.
|+| Looking for .classpath, Found it in BRANCH : 442862d4a8d4d18d0e176ded8795dd45a24528fc .classpath. Storing it in gumbler_testing/tmp/.
....
checking for .project..
|+| Looking for .project, Found it in BRANCH : 697fd66aae9beed107e13f49a741455f1d9d8dd9 .project. Storing it in gumbler_testing/tmp/.
|+| Looking for .project, Found it in BRANCH : 0941d9e0e0dda3ee1d9d4dda757d59ffb641abcf .project. Storing it in gumbler_testing/tmp/.
|+| Looking for .project, Found it in BRANCH : 47bdb537789c034493e94d8977eae77ecbfd5b24 .project. Storing it in gumbler_testing/tmp/.
....
checking for .settings..
....

.classpath or .project are not damaging in this case and, hence, are used as the example. On a pen test or in collaborative projects I have found much worse (cough usernames, passwords). This shouldn’t be that surprising.

Searching Commit Logs

Another use case for gumbler is to look through commit history. Using Ruby on Rails as an example, we can search from for any commit with “CVE” in it. Gumbler will output a diff from the files changed in the commit.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


$. ruby gumbler/gumbler.rb --grep CVE rails/ tmp/
|!| skipping .gitignore, searching commit log for CVE
$. ls tmp/
060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30_15:13:03_-0700.diff 88cc1688d0cb828c17706b41a8bd27870f2a2beb-2013-01-08_12:11:18_-0800.diff
08d0a11a3f62718d601d39e617c834759cf59bbb-2014-02-18_15:38:50_-0300.diff 8be6913990c30f63618173da722148892348dcc9-2013-03-15_17:45:53_-0700.diff
0b58a7ff420d7ef4b643c521a62be7259dd2f5cb-2011-02-08_14:21:12_-0800.diff 8e577fe560d5756fcc67840ba304d79ada6804e4-2013-01-08_12:41:24_-0800.diff
0c7ac34aed1845044cd1911e5a775366d7ca41c1-2013-12-02_16:42:16_-0800.diff 9340f89849606dba02f44038171f3837f883fd4e-2012-05-30_15:09:13_-0700.diff
2392535f4085d88186097e3c23414e958fb1d16d-2013-03-18_10:17:32_-0700.diff 93fb4c1e62dc9605eecbfaffda2becc85890fa5f-2014-07-10_10:20:16_-0700.diff
...
$. cat 060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30_15\:13\:03_-0700.diff 
060c91cd59ab86583a8f2f52142960d3433f62f5-2012-05-30 15:13:03 -0700==> 2012-05-30 15:13:03 -0700 Strip [nil] from parameters hash. Thanks to Ben Murphy for reporting this! Strip [nil] from parameters hash.
Thanks to Ben Murphy for reporting this!

CVE-2012-2660

:100644 100644 aa5ba3e... 6757a53... M actionpack/lib/action_dispatch/http/request.rb
:100644 100644 c3f009a... 6ea66f9... M actionpack/test/dispatch/request/query_string_parsing_test.rb

diff --git a/actionpack/lib/action_dispatch/http/request.rb b/actionpack/lib/action_dispatch/http/request.rb
index aa5ba3e..6757a53 100644
--- a/actionpack/lib/action_dispatch/http/request.rb
+++ b/actionpack/lib/action_dispatch/http/request.rb
....

As the README says, be careful using the tool as it uses Command Execution to search. A malicious git project could take advantage of this. Ping me with better ways to handle this.

IPv6 DNS Guessing Notes

Tue, 19 Aug 2014 00:00:00 +0000

A hostname with an IPv6 address is stored as a AAAA resource record in DNS (see AAAA record). There are many DNS hostname bruteforcing tools, personally I like Fierce. Suppose we have already run our hostname bruteforcing tool against a target domain (e.g. facebook.com). Below we use dig to do a AAAA record lookup for each hostname. Note, the DNS server we use matters. In this example we use 8.8.8.8, to confirm different results try using a.ns.facebook.com instead. Host can also be used instead of dig:

1
2
3
4
5
6
7
8


$> cat fb_hosts.txt | while read line; do echo $line" Results:" && dig @8.8.8.8 +noall +answer AAAA $line && echo; done
mobile.facebook.com Results:

ipv6.facebook.com Results:

www.facebook.com Results:
www.facebook.com. 1903 IN CNAME star.c10r.facebook.com.
star.c10r.facebook.com. 30 IN AAAA 2a03:2880:f00b:900:face:b00c:0:1

An offline/quieter way is to use the DNS Record (ANY) set from the Internet-Wide Scan Data Repository done by Rapid7. Using facebook.com as an example again:

1

$> pigz -dc 20140310_dnsrecords.gz | grep -i "\.facebook\.com" | grep AAAA

This didn’t turn up very many results but we can combine the two:

1
2
3
4
5


$> pigz -dc 20140310_dnsrecords.gz | zgrep "\.facebook\.com" | grep ",A," | cut -d"," -f1 | while read line; do echo $line" Results:" && dig @8.8.8.8 +noall +answer AAAA $line && echo; done
...
z.c10r.facebook.com Results:
z.c10r.facebook.com. 59 IN AAAA 2a03:2880:f00b:305:face:b00c:0:1
...

You get the idea =). Not a new concept or technique, just wanted to put some notes in one place.