My primary purpose for DumbyLand is to build up a sandbox for simulating different types of IT environments. A significant part of this is collecting logs for analysis to build up my blue team skills further. Architecting, engineering, and deploying SIEM and some basic controls are a huge part of that.
I want to be able to play around with different technologies, but to get things rolling, it’s best to just choose 1 thing at a time that sounds fun and interesting to set up. Graylog is something I’ve been hearing a lot about, it’s definitely something worth checking out and learning!
Instructions to get going for Graylog seem simple enough! Download the OVA, import it, start the VM, and wait.
It starts out with a simple setup of 4GB RAM, 20GB disk, and 2 CPU. I dig this, much preferred to SIEMonster and others that demanded 5 separate systems to get started, most requiring 8GB RAM each! Not something I can do until I set my server up with more RAM for a few hundred more bucks… which I’m hoping to hold off as long as possible to see if this price war lets up.
Enough of that, though. The system got a DHCP lease, and I hit it from the web interface.
Boom! Simple as that! Login with default admin/admin, and we’re rolling.
1. Send in first log messages
Let’s do it! We’ve got a basic Windows environment already going in the lab, so let’s pull it in.
From reading that community page, I understand that nxlog is going to be the best option for forwarding logs, and I’m always down to learn something new. Unfortunately, Graylog just leaves us with a “they’ll pick it up from here”.
Over on the nxlog website, there are downloads for the community edition for Windows machines. Documentation seems solid so I gave it a pass through reading about features and the basic functionality of it. This will be great to review later in-depth for a good understanding, but I think right now it’s most important to just get it up and going to see what we’ve got.
We’re going to go with a basic agent-based collection (Section 3.3.1) which should require something running nxlog on the Graylog server but we’ll get to that when we need to.
RDP in to my Windows desktop VM, and then to my DC.
Since I don't want to abandon all good security practice, I’m not gonna go around simply disabling these controls until I have good reason to. To allow this download, I had to add nxlog.co to my trusted sites. Search for ‘Internet Options’, then to Security tab, Trusted Sites, and then Sites to add the URL for the agent. Phew! But the rest was pretty straight forward. Downloaded the .msi, ran it, all good!
After the install finished I should have expected the readme to pop up, but it didn’t. Back to the documentation! The default installation location for me was “C:\Program Files (x86)\nxlog” which contains the conf folder.
In the documentation (Chapter 33. Microsoft Windows) there is a section for advice/recommendations/setup for Windows logging. For now, I really only care about Windows EventLog which there’s a jump to (Chapter 80. Windows Event Log). From there, I jump to (80.1. Local Collection with im_msvistalog) which is used for any system running 2008/Vista or later.
They provide an example of collecting all Windows EventLog data in JSON, written to a local file. I dunno exactly where we’ll go from here to get it to Graylog, but it’s at least the right direction! So let’s do that. Back on my DC, I’m gonna leave basically all of the conf alone, and just add the snippets that are included inthe docs here.
I forgot to open Notepad with admin privs, but also realized that even though the basic config file is super basic I should create a conf backup… good habits and all. Alright!
Now that we’ve got some basic configuration which we can revisit later, let’s get it on (12.1.1. Installing Interactively) and we’re up step 5. to verify the configuration, which passed. Now I opened services by searching ‘Services’, opened it, and then started the nxlog service.
The test config points the json output to “C:\test\sysmon.json” which I hadn’t created the dir for before starting the service. Maybe that’s an issue in getting it to write, but not sure. I double-checked config compared to what was in the example, and then also pulled up the logs for nxlog itself (C:\Program Files (x86)\nxlog\data\nxlog.txt) and saw that I’m getting a message of “WARNING not starting unused module eventlog” which makes me think that even though I defined an input, it doesn’t know what to do with the information.
Sure enough, there’s also an additional error message “WARNING no routes defined!”. I took a quick look at an example from Loggly for nxlog https://www.loggly.com/docs/logging-from-windows/ and found an example route that I used to copy and make one for myself. They have a section “internal” that I’m not sure what it does, but is an additional data source to investigate later.
define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
# Check the size of our log file hourly, rotate if larger than 5MB
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
# Rotate our log file every week on Sunday at midnight
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
Exec $Message = to_json();
Path eventlog => file
This is my full conf for a test setup working. Hooray!! It’s very straightforward and simple. Eventlog module pulls down the information, and then uses the defined output ‘file’ to write it to a file all in the magic of json. We’ll see if I have any troubles but at least this is on the road to progress!
I need to configure my Graylog server with a static IP so that I can start to forward stuff to it, then I’ll configure nxlog to forward over the network. Back in the Graylog webconfig, I poked around the options, but doesn’t seem like it will be that easy, which is understandable.
They instruct you to go about it the standard way of editing /etc/network/interfaces and configuring there, as I’ll do now. Since you’re modifying the network connection, directly console in via ESXi and login with ubuntu/ubuntu, and you can follow the instructions below but it’s a standard configuration of the interface.
Logged back into the web config at 192.168.2.20, woohoo! This is all set. Now just to update the configuration on my DC to push towards this IP address.
Slight pivot on configuration. I’m having trouble with Graylog inputs. I get the general gist, but get these “failures” on trying to start the inputs. Not eactly sure what’s wrong, but going to attempt a new thing here.
This is a “content pack” which is a bunch of pre-configured stuff for Graylog, this one specifically for AD security auditing. It seems pretty damn nifty, and the Github includes a nxlog configuration to use. Import the content pack via Graylog webUI, and it will create the input and some dashboards and a couple other things.
Still having trouble getting this new input to work, I did what works best for anyone… I did a nice little restart on the Graylog host. Running ‘netstat -tulnp’ showed me that the designated port 5414 was running, but only on udp6. Perhaps this is just me being misinformed on how things should look, but I’m now the proud recipient of AD logs!
This is really getting my server to spin. It seems like Graylog is starting to choke up, so I’m gonna give it a minute since there was a sudden spike in activity overall and see if things normalize some.
And overall, my server is feeling like this with that
After giving it a while and still not getting a response when searching for messages in Graylog, I figure it’s time to look for more help, perhaps bumping up its CPU availability some.
I ended up not having to do this… yet, at least. I went back and reviewed the configuration steps, re-performed the basic graylog-ctl stuff, and then did the reconfigure command. I went back to the web UI, did a search against the stream from my Windows log input, and…
We have liftoff! Whoooooooooooooooooooooooooo!!! Now that we have a SIEM up and going, I can work piecemeal to have them ingested into Graylog as I bring services and additional machines online. This is so very exciting for me!
Next up I’d really like to be IDS, but I’m really not sure how I’m gonna get traffic mirroring set up on my box since I don’t have vCenter stuff rolling yet. I COULD just set that up, but that seems like a lot of effort for a single (important!) feature of this lab.
Also interestingly enough… now that I got everything working and am running queries against the data, my CPU usage dropped dramatically! It really seems like something got screwed up in the configuration process and the reconfiguration kicked everything back into gear.
For a minute I was really concerned that I was already making my box choke! WIth just 1 input into the SIEM and no heavy queries or anything, already maxing it at a sustained 90% leaving my box at just shy of 50% utilization had me scared. Glad to see things are REALLY starting to work out! Even at the most desperate of times, my server hit a maximum load of 57.4% and an average of 37.44% over the last hour where the vast majority of the time it was in that CPU % funk.
I think that’s pretty god damn impressive, if I may say so myself. To review, this is currently what we’ve got running:
Hypervisor Host - ESXi (Supermicro E300-8D)
Opnsense - Core Router
192.168.1.1 / 192.168.2.1
Windows 2016 - Domain Controller
Windows 7 - Client Machine / Admin Machine
Windows 7 - Victim Machine
And let’s keep going! Since things are working well right now, I grabbed a snapshot of Graylog as well as my domain controller. My progress so far has been quite pleasing!
Reviewing OPNsense configuration again real quick, I was reminded that it has built-in IDS features! Doing light digging, I discover it’s based on Suricata, cool!
Let’s try it out! To make the most use of the work I just completed, and to have a bit more fun, let’s also get OPNSense set up with sending its logs to Graylog. This is considered “remote logging” in OPNSense terminology and to be honest was a pain in the ass to find proper documentation for… I never found any.
To get started, navigate to “System > Settings > Logging”
Seems like it should be straight forward, but let’s see!
I have configured Graylog with the following content pack for PfSense since it should work just fine for OPNSense
Naturally… things aren’t straightforward. Now my Graylog isn’t getting Windows logs, what the fuck! Back to basics with troubleshooting this. Everything seems good with nxlog configuration, and the DC can ping my Graylog server. Given my inconsistency earlier, I’m gonna try to just restart Graylog and see what happens.
Back to inputs, pull back all messages for Windows, and sure enough… there they are! There’s definitely some inconsistency going on with my set up. I believe it’s likely Elastisearch since all the data going back over an hour is suddenly available in the UI to search for… bah! Oh well, the joys of simulating and labs!
Guess what else we’re getting now! That’s right…
Woot! That content pack has a Pfsense-Logs input with extractors for a few different things. It pulled out the authentication logs in opnsense with a good source, so I think that should definitely satisfy my basic needs!
This screen makes me happy. Very happy indeed! Quite satisfied with how it’s handled the load so far, even if some things have been a bit funky. Maybe ECC RAM is gonna be a help? I’ll spend a day on stability and load testing in the future once more things are running.
Now that I've got a SIEM and IDS features to play around with, I can start to bring on additional services and expand capability and scope! I have plans to bring on representations of development and production servers. In the meantime I also really want to bring up some vulnerable servers to play around with pentesting, maybe try out some brute forcing fun, throw up Mimikatz as well as some Icebreaker (https://github.com/DanMcInerney/icebreaker) and other MITM goodies. DNS poisoning, RAT, some general C2, data exfil... all that's good stuff I'll be getting to as I build this out!
|define ROOT C:\Program Files (x86)\nxlog|
# For windows vista/2008 and above use:
# For windows 2003 and earlier use the following:
# Module im_mseventlog
Path in => out
I have no honest clue how to really do Windows. While I have worked extensively with Windows in my life, I’ve never been a true systems administrator for it. I haven’t built active directory from nothing. This is going to be me documenting my stumbling journey through the wide and wonderful world of Windows environments! Let’s just jump straight in.
Let’s start with about as basic as you can get. A standard Windows 2016 server, and then a Windows 7 workstation machine.
We’re gonna get rolling right with the domain controller.
Next let’s get a PC up and going. This is going to be my primary interface into the environment. I am going to use it as if it were my client machine for administering the Windows servers and everything else.
This is just a plain ol’ Windows 7 computer at the moment. Let’s get it hooked up into the domain, and create myself a user and domain admin, shall we? For now, I have to do a console authentication directly to the server until I can get this client machine hooked up to the domain
Now, let’s pull up the Active Directory Administrative Center
Cmd.exe > dsac.exe
On the left side, I select dumbyland (local) > Users > New > User
And they’re created! I feel like we should next get going on the client machine. Here’s a vanilla machine, and I have no clue how to do this right… but let’s figure some stuff out.
In the System Properties window, there is the option to “Change” domain settings. Select that. It will prompt you to enter credentials -- I used the dumby-user account I just created to simulate a new, regular user!
Okay this just isn’t working. I have no clue what else I need to do. I update my DHCP settings to ensure that 192.168.2.10 (domain controller) is the primary DNS since at one point this was giving me DNS error of some kind. Now I’m getting a logon failure error
I figured that perhaps the issue is that I was using an account that I hadn’t yet logged in with before. To get around this, I used the primary administrator account to sign-in for the first time. My prediction seems correct… and I signed in for the first time! This message was a beautiful sight.
After this, I reboot the PC, and RDP back in to it. I still can’t sign-in as the normal user, but we’ll get that figured out soon enough. I replaced this dumby-user account with the administrator, and successfully signed back in.
Well, this is at least giving me hope!
I can’t get a sign-in to the regular user to work, but I’ve got a canvas to start from. My main goal right now was direct administration of the domain controller since I’m doing everything through the ESXi web console -- not exactly speedy and reliable! There are also some issues with directly RDP into the DC, so I’m gonna be using my Windows 7 VM as a ‘jumpbox’ into my domain. (Laptop RDP > Win 7 VM, RDP > Server 2016 VM)
RDPception! Here is my laptop RDP into the Windows 7 VM. From that, I RDP into the domain controller VM. Hooray!
For some reason now that I have direct access, password change and logon for the user worked. I opened up a new RDP session, signed in, and was prompted to change the password.
To get away from the default administrator account, I’m going to start using this new administrative account. First things first, it’s complaining about having Remote Desktop Users permissions -- let’s get that going right now.
I opened up my existing admin RDP access through the built-in Administrator account, and right-click my dumby-admin account, and added it to the ‘Remote Desktop Users’ group as well as the ‘Domain Admins’ group. I then signed back in with my dumby-admin account, and kicked out the other RDP session. Hooray!
And now I RDP back into the domain controller with my dumby-admin account from my client PC. WHOOO!I absolutely love hitting these milestone points in little projects like this. Another quick update was to add my user account to the ‘Remote Desktop Users’ group so that I can access it from my laptop directly using my client account.
For some reason… this isn’t working. I’m not sure if there’s a delay in it taking effect or what, but doing a domain sign-in to my ‘dumby-user’ account via RDP is still giving me the permissions error, whomp.
What I’m reading right now is that this is because of permissions on the local computer I’m trying to access, since I need to update its own members of the “Remote Desktop Users” group. It’s worth a try! Let’s first make the change directly, and then if that works, I’ll get some experience with Group Policy and distribute that change to all future computers as well
Okay, so first things first. Search ‘users’, and open ‘Edit local users and groups’. Go to ‘Groups’ > ‘Remote Desktop Users’. Select ‘Add’, and then I searched ‘dumby-user’, which then automatically discovered the domain user. This all makes perfect sense now! You must give individual permissions on each endpoint of which users can access it via RDP, but that user can be a domain user. Good to know!
It doesn’t seem like I needed to do these permissions for my admin account… perhaps because it’s a domain admin? An interesting note, and if that’s true, it prevents me from needing to do any GPO stuff for now, it seems, since I’m only worried about my own user account being able to access this VM. Hooray!
Now, I have a domain authenticated user machine! From it, I can use it to access the DC via my domain admin account -- dumby-admin. This is how I was hoping for it all to work! Very happy with all of this so far.
Performance of my server during all this….
Everything is super snappy. RDP is working fantastically -- console works a LOT better since I did the driver optimization fix for storage.
To start having some real fun, I want a 2nd client that can act as a vulnerable machine. Let’s see what trouble we can get up to! To make things easier, I figure it’s worth learning how to clone VMs with ESXi. I found this guide to do so:
And that’s basically it! I repeat my instructions above to get this new machine joined to the domain, and now I have a nice little set up going! Next up is to introduce some logging, and then I think I’ll have some fun setting up attacks and reviewing that information.
The last few weeks have been extremely difficult. Let alone the last year... I haven't done nearly as well of a job as I would have liked on keeping up with my blog, but I'm not short on topics to discuss or work to share! There was a stimulus that kicked me into high-gear project mode. Given a choice, I wouldn't ever choose to go through it again. With the passing of our hacker brother d3c4f, I hit that high-gear mode. I gotta step up, I gotta do more, and I gotta contribute back to the community I hold so dear.
To this end... I introduce...
A home lab is a crucial component of the modern IT professional, and especially so in the security field. By nature, security is cross-discipline and requires a diverse skillset. Unless you've got 10 years of IT experience, it's highly unlikely that you can cover all the ones directly important to your specific position or specialty. This becomes a fundamental issue for both those experienced in the field looking to expand their teams, as well as those new to the field who are finding their way in.
I know this first-hand. I've got a solid 8 years of professional experience, and I'd say about 6 of those were strictly IT. In that time, I never got to administer or experience an enterprise environment. I had little exposure to hardware, or complex configurations. I hadn't had a reason to understand a full-blown virtualized environment. Beyond a home network, some small IT shop deployments, and personal projects my systems administration skills are quite weak! I've read Windows logs, but I don't know exactly what stimulus leads to which logs -- being able to directly simulate a user failing to login, or brute forcing a Windows admin account is something I am in dire need of at my current stage of career.
So let's get to the meat!
I needed a lab. I know that with my personality and skill level, I would only ever fully utilize such a lab if I were able to take it around and share my experience, working with others to accomplish and review the work. Portability was key! Naturally, the market solution to that is the NUC. Extremely capable, extremely transportable.
Recon Infosec created a mobile DFIR lab, and went this route. 3x Skull Canyon NUC, and 1 more machine I imagine is their primary network interface. The NUC are sleek packages with passive cooling, but i7 and 32GB DDR4 RAM. Quite a powerhouse, and you can slap 3 into a nice little case ready to go.
And doing what I do best, I leaned on others for further information.
And, naturally, some jerk got me sallivating for something I had no clue even existed! Thanks @M3atShi3ld you jabroni. He casually threw this out there... and I instantly went into a downspin thinking of all the possibilities -- of every possible pro/con I could wrap my head around.
Come 05/02, and I'm speeding home from for work after receiving delivery confirmation of a very sexy box.
There's an E300-8D in that box. So, why did I make the choice for this box?
There's a few things that led me to choosing it.
It's smol. Quite smol.
Initial access was quite straight-forward. RTFM. Remove 2 screws, slide cover plate back. Expose all the juicy inners!
Let's break into all this fancy new hardware.
Timetec Hynix IC 32GB Kit (2x16GB) DDR4 2400 MHz PC4-19200 Non-ECC Unbuffered 1.2V CL16 2Rx8 Dual Rank 288 Pin UDIMM
Samsung 860 EVO 500GB mSATA Internal SSD (MZ-M6E500BW)
WD Black 1TB Performance Mobile Hard Disk Drive - 7200 RPM
Samsung 860 EVO 500GB M.2 SATA Internal SSD (MZ-N6E500BW)
Total Build Cost: ~$1300
Start with RAM since I know you can't screw it up, take the little wins before you end up in wars of attrition over stupid things like moving the location of a single stand-off so you can use the mSATA drive.
So about that. Out-of-the-box, I'm not sure what they intend for you to be able to use the stand-off for. It's right in the path of the mSATA PCI slot, and manual indicates you should move the stand-off to secure the mSATA drive. Okay, whatever.
No. Not just whatever. I'm going to save anyone else who builds this a lot of time. Tear it down -- all apart. You will be removing the motherboard to move this standoff. Just heed my lessons.
Like any good hacker, I had it physically disassembled and in pieces within an hour of being hands-on with it.
Next up was the M.2 drive, and the 2.5" drive. I wanted at least something to have a good amount of storage for cheap, long-term storage. Originally I had read there was a 4 TB drive, and ordered it... but it was a 3.5" drive. Ugh. WD Black was a striaghtforward, super cheap option, but you could go with another SSD or maybe find a bigger 2.5" drive.
I decided that since I had a free license thanks to .edu hookups, I'd go with "real boy" virtualization and straight to ESXI 6.5. So let's get it going! Now this part should be really easy if you're not a complete idiot like me. I struggled for way too long and too many USB drives, ISO images before I finally got the right god damn ESXi installer image on a USB. Plugged it in, and wait for the first satisfactory boot!
First ESXi boot!
My first connect up to it was me hooking Ethernet cable up to a router I had laying around, and then my PC up to the router. This was my first method of connecting up to the management port on the server, and worked this way for a little while until I discovered direct connect thanks to a little magic called Automatic MDI Crossover -- more on that later!
For some reason, I had a deep-seated need to change the hostname first thing. So I went and did that!
I set it by going to the following setting:
Networking > Default TCP/IP stack > Edit Settings
At first, I did something really dumb by screwing around with hardware passthru which was due to me just being really misled with my thinking. I eventually turned back around that rabbit hole. Took a full system reset, but I was able to get back in and screwing around with configuring it.
Now for the proper setup --
Set a static IP on the primary management NIC -- vmnic0. I chose 192.168.88.50
Set a static IP on the Ethernet interface on your computer (a USB adapter in the case of my laptop) -- 192.168.88.100
Connect Ehternet cable from computer to vmnic0
Access the web interface from computer -- https://192.168.88.50
Appreciate the great success!
With all this setup, management is that much better. Now you only need to have any regular old cable and a way to connect them, right up to the server!
But this only covers management networking. Given that my ultimate purpose is to travel this thing around, and share in the experience... I need a way to connect up to the internal networks, whatever they may become. There's also questions here of how exactly I'll bridge those connections, and how I'll hook people up to it on the outside. Lots to learn!
I also have some dreams of it being used in situations where I can utilize it as a drop-in IR box, or something fun like that. The box has a lot of physical networking capability... and let's cover all that!
So, out of the box it has 6 physical NIC on the motherboard.
vmnic0 - Already assigned to management for ESXi
vmnic1 - I wanted to use this port to provide an Internet uplink for the box
vmnic2 - And this one for an external, physical LAN
vmnic3 - Gaming (No plans yet, figure I could eventually use it for mobile LAN server)
vmnic4 - IDS
vmnic5 - Unallocated
At this point, I'm not precisely sure what I wanna do with the rest of this, but have some general ideas. Let's start to implement at least some of those ideas on the box itself! We'll now dig into vSwitch networking within ESXi.
I have a general idea what to do, and this is my playground. Let's go for it!
We're going to create a series of vSwitch to represent different networks... obviously. 1 for VMs, 1 for physical access, 1 for Internet/"Gateway", etc. When you create a vSwitch, you can also designate a physical uplink port.
There is a default switch -- vSwitch0 that gets created with vmnic0 set as the physical uplink -- that's how the management network works! So let's create a vSwitch called "VM Access", because I need an interface into the server for physical devices to get in. I wanted to save the 2nd NIC for gateway since it's more of a "management" type service, so we'll use the 3rd physical NIC for that switch.
Name: "VM Access"
Port Group: "Access VMs"
And that's that! Now, you've assigned physical NIC to virtual switches... but need to add VMs to those virtual networks! Network interfaces to VMs in ESXi are called "Port Groups". You create a new Port Group, and link it directly to a vSwitch. Let's create a couple more of the vSwitch, and also some port groups to use for the VMs. I did not have the best naming conventions here by any means, do better than me!
Port Group: "Internet Access"
Name: "VM Network"
Port Group: "VM Connection"
Port Group: "Management Network"
We've got a physical NIC as figured out as we can so far, let's get the virtual side up and going!
Now, as I mentioned before, I wasn't entirely sure how this was all going to work. I had figured that a router VM of some kind would be needed, and that sounded fun enough to play with anyways! I downloaded a copy of OPNSense, and set up a VM for it.
We're going to initially have 3 interfaces for this VM:
Internet Access - Provide Internet gateway for the router
VM Connection - Provide VMs with a connection to the router
Access VMs - Provide the physical LAN access to the router
This next bit was a lot of fun. It took me 2 laptops, a few afternoons, and some freetime at work to completely figure this all out... but boy did I learn and do a lot to get there!
With console access via ESXi web client, I was connected to the opnsense VM. There are very limited options from this point, and I figured that it would be straight-forward enough to figure out a few interfaces. Boy was I so, so wrong with my ideas and it caused me a lot of grief. To be fair... the problem I was experiencing was *extremely* basic, and nobody that I asked for help had any for me, so HAH!
I went ahead and configured the 3 interfaces as I saw them. During basic configuration, you set up 2 interfaces -- WAN and LAN. There is also a 3rd, simply called OPT1 (Which I later re-named to VM1, as indicated above). Neat little tidbit about OPT1 and LAN -- LAN gets rules by default to allow traffic. OPT1 has zero rules to allow traffic by default, and you can't do jack shit from it until you configure firewall rules on it. The router was configured to provide DHCP for both interfaces.
I did not know about the firewall rules.
I also did not realize how painful it is to swap back and forth from static IP on 1 physical NIC, over to DHCP on the other physical NIC in vane attempts to figure out what in the fuck you screwed up on the configuration.
My laptop connected up, got DHCP -- everything looked right. IP was good, subnet good, gateway good. Could not ping the router. Could not access the webconfig. WHAT THE FUCK! I know it's pretty common that firewall/routers won't have ping enabled by default, that made sense. But no SSH, and no webconfig... annoying and weird!
Here is my Saturday, hacking away with primary laptop on my left knee, and my Chromebook on the right -- both without Ethernet ports, and me struggling to rotate through my stock of 4 sort-of misbehaving USB-Ethernet adapters. It was a great time! I ended up using a LANTurtle, and a simple Microprice adapter I had laying around. The LANTurtle was great and worked reliably, except it has its own layer 3 abstraction that made troubleshooting *really* hard to pindown and be really sure of myself.
So, here I am. Swapping cables in/out, adapters in/out, swapping back and forth from ESXi management, and connected to the VM Access network. Using tcpdump from the router console, I can see pings and web requests coming from my laptop, so I know layer 1 is working, but SOMETHING IS BLOCKING SHIT. I'm going nuts, absolutely convinced that it's a VMWare firewalling issue, searching everywhere and asking people for advice.
I finally broke down and went with a new tactical route -- from the virtual side. During a free moment I had uploaded a few Windows ISO and created a Win7 VM. I boot that up, and it connected to the VM Network vSwitch. Access it via the webconsole. DHCP connects, I get an IP address on the 192.168.1.x range (the LAN interface on the router) and... AND... IT PINGS THE FUCKING ROUTER. AND CONNECTS TO THE WEB CONFIG OHMAHGAWDTHEFUQ.
Now another REALLY fun bit... This was Win7, with old ass IE being the only available browser. OPNSense webconfig doesn't work for a god damn thing AT ALLLLLLLLLLLL in that version of IE, naturally.
Another REALLY fun thing about ESXi is that there is no good way to upload files to the VMs. No drag'n drop like I'm so used to with type 2 hypervisors. The best method I found was to create a data ISO, upload that to ESXi, and then connect that up to the VM as a disk drive. Access the ISO, and pull off the data.
After a few flip flops back and forth with insane ideas of what was wrong, I did the work and signed up for a trial with Daemon Tools Lite for their data ISO creation app, loaded one up with Chrome (remember to use the OFFLINE installer if you haven't yet figured out WAN, like me at this point), and got that onto my desktop VM.
Chrome opened up to the OPNSense webconfig beautifully! Now I'm in business. Oh yes. First thing I did was switch around LAN and OPT assignments for the interfaces.
Back on my laptop... I CAN PING THE GATEWAY AHHHHHHH. And I can access the webconfiAAAAAAAAAAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHH
Now like any good systems administrator, first thing I did was go into the firewall, and add ANY*ANY rules to all the interfaces. Back at the Win7 VM, I can ping the gateway. I can ping my laptop. My laptop can ping the gateway, it can ping my VM. Everybody loves everybody. Everything is good in the world.
*Note I did make notes of stupid security things I did. I honestly kinda like this as a retrospective of how certain mistakes are made!
Here we'll set up some good portability for this mighty fella. I already had a nice, padded case from Harbor Freight that I was using for my poorly abandoned SDR gear. Now repurposed for real use, this package is coming along smoothly!
Another big part of that is getting the hell away from those damned USB adapters. I made a decision to go extremely basic with my initial rollout of network gear.
1 TP-Link gigabit 8-port switch
1 ZyXEL 802.11n Wi-Fi AP [NWA1121-NI]
I'm not entirely sold on either bits of this gear, but so far they have served me well! Connect the first and third Ethernet ports on the server up to the switch, and then the AP to the switch. Cabling is a mess, and ideally I'll do PoE to an extremely small AP, but I couldn't find any of the Mikrotik MiniAP available, and didn't want to totally invest in a PoE switch as I wasn't sure feature set and expandability required.
Who knows, mabye I'll get an itch or financial support to go crazy and use the onboard SFP+ ports! Those mean all sorts of applicability to being dropped in for use in all sorts of environments as they are 10gigabit uplinks! Just the gear for them isn't quite... consumer/hobbyist researcher friendly.
AP configuration was pretty straight forward. Set an SSID, connect up to SSID, it gets passed through to the switch. Switch passes it to the 3rd physical NIC. That gets passed to the core router VM. Now I just connect it up, make sure the VM is running, and I'm ready to roll via a simple Wifi connection!!! This should support quite a few cliens with plenty of speed to access VMs, hassle-free. Hooray!
One final convenience -- accessing the ESXi management from Wifi. Another entry into the "security todo" list, I created a new port group - "Management - From Wifi" and attached it to the "VM Access" vSwitch.
The ESXi host itself gets its network interfaces for management via "VMKernel NIC". The default one is attached to the default vSwitch. This one, we're going to attach to the "VM Access" vSwitch, as just mentioned. Now the ESXi host has access to the vmnic0 physical NIC for management at 192.168.88.50, and then over the Wifi/ physical LAN at 192.168.1.3. If I'm ever concerned about locking things down -- a red vs blue situation, or just around more untrusted/unsavory people, that interface can just be disabled or hardened. Now I can access VMs, and the ESXi host from the comfort of my laptop simply connected to a Wifi connection.
HOME STRETCH OF ALL THE FUNDAMENTAL BULLSHIT!
What a journey, am I right? The final goal of mine to have really "figured it all out" as far as the basic infrastrcuture goes was an Internet gateway connection. The reason this has been so difficult is all the networks around me are hobbled together with scotch tape and hope. I'm basically all Wifi at home due to a poor location of coaxial drops.
I found a way to deal with this via my handy dandy Pineapple Mk 5! It's a very capable and convenient device to keep around as an awesome wireless bridge or general router/gateway. I connected up to it, hooked it up to my home wifi. Then I connected a cable up from its Ethernet port to the server's 2nd physical NIC -- vmnic1, Gateway! IT'S FINALLY HAPPENING!
Checked out the opnsense webconfig, and what did I know... WAN IP ACHIEVED!
Connected to the "DumbyLand - VM Access" SSID, I can ping google.com, and I can access the ESXi web client, and I can ping my Win7 VM! I can access my online Docs! My VM can access the Internet! IT'S ALL WOOOOORKKKIIIINNNNNGGGGG!!!!
I've already messed around with a couple ideas for labs. With all of this figured out, the rest should be figuring out the VM environments themselves! I can bridge the worlds, and provide the wealth of the Internet to them.
For now, I think that's a fucking great start.
I love ya, d3c4f. We all do! And we'll sure miss you. You've inspired many, and will always continue to do so.
I might break a lot of minds with this first statement --
Security can be your first career. But this isn’t the norm, and you have the responsibility to pave your own path.
There are an incredible amount of ‘how to get into infosec’ resources out there. I’ve explained my own story, given it for others to integrate into their own talks about it, and read countless pleas from those who are lost.
There’s a reason for it. There is no path. We are finding some right now. It’s an incredibly daunting, but also awesome time for security noobies, and we are building a new future. I am on the frontlines, middle of the crossroads. I bust into the scene when it was a kindling mega-industry, and now it’s at the forefront of most people’s minds. Incredible times!
I’m gonna repeat this little bit again -- You have the responsibility to pave your path in information security. You’ll wander through the roads of many others. We’re paving over them, but can’t stamp them out. Today’s conversations have been raging for 20+ years and require a lot of context and knowledge if we want to contribute to them.
One thing we know is that there is a ‘talent shortage’, cybercrime isn’t going anywhere, and people are having a hard time finding their path ‘in’. I’d like to say there’s a solution. A workshop, certification, bootcamp, etc. but the reality is that it’s way too big of an issue for it to all be addressed in any one way. Enter the entropic world of ‘beginning’ security.
Security is a dangerous term. It speaks for a LOT. It means infrastructure, development, psychology, technology, science. Any human interest can expose an avenue in enriching a deeper security program. You have to understand complex ecosystems. Both highly technical, but maintaining a grounded reality in how systems are (ab)used by end users, what business priorities are, and making tough decisions surrounding risk exposure.
If you are unable to accomplish meaningful thought regarding the above tasks, I’d assert you will have a difficult time getting into security. You likely need more experience. It may be possible to go a strictly technical route, and it can certainly be fruitful for many people. I find that after so long as a bug bounty hunter, or pentester (separate from red teamer, IMO), people realize they hit a wall in those accompanying skills and lose satisfaction in their careers -- you’ll hear the term ‘soft skills’ a LOT from those who help build up career newbies.
Exposure is everything. If you haven’t been the user of a system, or been supporting them, it’s likely impossible to know their mindset. If you haven’t administered a Windows environment it’s hard to know what its logs mean. If you haven’t worked as part of a development team, it’s hard to have a grounded reality in how bugs become present, backdoors don’t get removed, etc. This doesn’t justify the problems in the first place, but you can’t create solutions without understanding the problems.
On the point of exposure, I am constantly asked what I’ve done to learn what (little) I realistically know. And my answer is… stuff. I’m not oblivious to the good fortune I’ve had with getting some of the strange jobs I’ve had. To give an idea of the exposure I’ve had, I’ll describe my professional journey.
First job ever was a bagger for a grocery store. I actually really enjoyed this experience, and got to see functions in lots of different departments over my 1.5 years doing it. I’m a HUGE fan of systems development, and seeing how people work within them. Grocery stores are an amalgamation of independent workshops that all have to work together. Produce and dairy are extremely different departments with their own management, workforce, inventory systems but there’s always crossover and cooperation with it and, say, the deli -- a whole other independant workshop.
By the time I had left high school, I had an opportunity to assist a mentor of mine. He had taken over IT operations for a small law firm, and needed some help during the daytime as this was a side gig. This was my first exposure to a computing environment outside of a home PC. It was active directory, Exchange, phones, computers, laptops, etc. While only for 10-15 employees, I got a ton of exposure and was left to my own devices when I probably shouldn’t have been… but I made it through without bringing the company down!
Life kicked in at that point and I needed something real. I went through a couple call center jobs for the next year or so. As much as I despised the work, I appreciate the exposure. This was a massive volume, highly technical, gigantic moving machine. It is crazy impressive these systems are boiled down to the point that anybody off the street can be trained to point, click, and navigate through what are extremely complex tasks. As soul-sucking as the work is, from a technical perspective, call centers are crazy impressive systems to me.
Things Get Strange
Now begins a strange string of experiences. At this point, IT was still the most interesting technical field of interest to me. Science, construction, heavy industry just hadn’t found a way to appeal. IT was it for me! Being sick of the grind, I paved a way into IT. I had expressed my interest in social circles, and got a call from a brother’s buddy. He was doing some contract IT gigs with a buddy of his, and they were getting a bit more serious. They needed an office manager.
So there I was. In the middle of a podunk IT shop in some random, poorly traveled mall. The company owner was a technician himself, with anywhere between 5-10 other technicians working in an IT contracting platform to find work. I ended up being responsible for managing their work by finding, negotiating, accepting, and designating contracts. They were out replacing CRT monitors in cook rooms, running cables to 5th floors of buildings, wiring up new door sensors, etc. They were making good money, and I was getting just that much more insight into what the real world of IT consists of.
These guys were idiots when it came to technology, all respect to them. I was training them on how to do their job regarding technical subjects as some kid locked away in a repair shop. My skills were greater than this, but I wouldn’t say I deserved more. I also made some REALLY cool Minecraft buildings in my time there!
Somehow Know Some Things, See Some Places
So, how did I even get to this point? I’m honestly not sure. I’ve always just screwed around with things. I’ve never been a huge fan of coding, I never see myself becoming a programmer. But I’ve done a bit of it. I had set up simple service servers, or hosted a website. I was pretty handy with fixing my own stuff, and could understand systems pretty well. That’s basically it. I wasn’t writing programs, connecting legacy equipment, setting up full services, none of that stuff. I had taken a Linux+ course in high school, and a Network+ class. With both, I had at least some level of requisite knowledge before starting the classes.
I had no bragging rights. I had never created a cool program, or built a service. There’s technical ability in youth that blows my mind -- look at the Minecraft host debacle. Those are all kids! It’s awesome, and scary! You can do better than me. You can have accomplished, and seen more, than I ever had by this point. I’m truly an idiot in so many ways.
But I’ve always been good at generally figuring out systems and finding a way to provide value in effort. Hands-on, I was hardly doing technical work outside of some laptop repair, or walking technicians through stuff on the phone. It was largely administrative. Creating spreadsheets, finding programs to track time and work, verifying payroll and staying on top of the techs to make sure reputation was upheld. Those are all things I would probably never think about without having had to do it myself.
And that’s a reality. People are having to do this stuff every day, without really knowing how. And it’s hard. It really, really is. No matter the platform or program, it will always take a level of simply ‘figuring stuff out’.
This became extremely obvious at my next gig. Eventually I got tired of the work, and I had a falling out with the owner. There was little integrity in what he was doing with his workers and I had no faith he would treat me better. I found an extremely odd gig taking a 6-month contract going around to State Farm offices and doing very basic tech work. It was replacing a server, the user PCs, a printer, as well as setting up a new NAS device. That was it!
I got to travel all over the state and work with agents who lived in the middle of the city, to rural cowboys with a tiny office in the corner of their basement who know all their clients by name, and see them daily. It was a TON of diversity but they all got the EXACT same systems, to a tee. This was very eye-opening and again, a wonderful exposure to the world!
Eventually the contract ended, and a decent paycheck for fun work came to a close. I got hired at a MSP (Managed Services Provider) doing entry level engineering/admin work. They focused on small government, educational, and medical clients. There was some interesting HIPAA stuff (I built a custom music streaming server for a client), but overall standard IT work. I only worked here for 4 months before being viciously thrown under the bus and voted off the island.
Browsing local job posting sites, I came across another odd opportunity. I interviewed for a customer service position at a local technology services company. They specialized in providing wireless equipment to large organizations, ISPs, ski resorts, etc. but also managed the business center for a local ski lodge and had developed a CCTV system for a remote mountain estate. I never answered a single phone call or email as a customer service worker.
Somehow I was immediately dumped into doing… engineering stuff? To be honest, the vast majority of my days were filled with me reading Reddit, forums, blogs, etc. because it was such a poorly run show. But, again, wow, the exposure! I did everything! I broke into that CCTV server since they lost the root password, and mapped out a diagram of their security cameras.
I created a PBX system to replace the phone system for a real estate office. Afterwards, I was roped into an 8-weekend project completely redoing the entire network for said office. There was also a contract to manage remote gateways for digital billboard companies, and we were responsible for uptime. So I got to mess around with Icinga (WHAT A NIGHTMARE OMG), and climbed a few of the billboards to mess around with their electronics. That was awesome! I met a guy who ran a wireless ISP and learned a TON with him, basically all of my early hands-on Linux was with him.
One interesting note about this workplace is that it happened to be a feeder for 2 of the best IT guys I know in the area. They hadn’t worked alongside each other, but very nearly did so, and both turned out to be amazing parts of the community. The company was in the grips of a death rattle when I was working there (extremely poor management, and a little bad luck), but it was an incredible shop in its legacy. I was given too much responsibility, but tried my best, and will always be thankful for the opportunity.
Experiencing The Boring World
Last but not least on my pathway here, is the QA work! A local company completely develops their own home automation gear -- from the hardware on the controllers, to the drivers that bring in 3rd party devices. It was an INCREDIBLY interesting organization as it included absolutely everything from top-tier networks and technology in their backend, to supporting the users in their homes being able to simply turn on/off their lights at a switch.
There was a web services department, hardware QA, software QA, product QA, software development, in-house hardware engineering, marketing, sales, a full external training department, customer service, and dealer support. Each piece of technology was siloed. 1 group would work on the drivers for A/V gear, the other for lights and switches. One controlled all the media management, metadata lookup, all that fun stuff. But that had to tie into the backend of the service delivery, where web services came in. I got to work a dual role between deep in Linux and the product itself, and being a QA resource for web services.
Web services, as a term, meant absolutely nothing to me. The entire time I worked there, I was googling the definition to make sure I understood what in the hell was going on. I hate ‘developers’. There were maybe 2 developers on that whole team -- the rest were coders. They took no satisfaction in the systems, understanding the underlying tech. They wrote their own code and pushed it, everything else be damned. That was a demoralizing aspect, but a true exposure of reality -- people don’t have the will/time/ability to care about all of this stuff, and it’s okay! But don’t call yourself a developer… ugh. Rant for a different time!
One significant aspect of that team I got to deeply understand was TURN. These are systems that go into people’s homes. The systems are installed by dealers, using whatever infrastructure the end customer ultimately has at their home. Dealers, at the end of the day, are chasing a buck and can’t be relied to truly do things in the best interest of the customer -- such as configuring firewalls.
This I appreciated. Instead of relying on the techs to be able to configure the network properly, they bypassed it. These systems were super capable and obviously, people aren’t always home. There was a significant need to access them remotely to toggle lights, set a thermostat, configure a note, check on cameras, etc. So no firewall rules necessary! You traverse it. By the home controller setting up an initial connection with the company’s backend, there was a permanent channel that enabled poking the home equipment directly without having random holes in a firewall.
It was an extremely complex and awesome system. I still don’t understand most of it, but got to sit in a cube with the programmers a few times and see how they operate. I got to read & understand a full RFC, configure AWS buckets, work with programmers, see the development process and work in a gigantic, international corporation. Lots, and lots of reality experienced there. PQA was pretty straight forward -- pushing buttons everyday, poking around Linux, running test cases. It was largely boring work, but I got to see the underbelly of a beast! Super cool!
And that gets us to today! Phew!
Outside of my professional experience, there is another significant aspect into breaking into the security field.
In 2013, after years of looking bright-eyed at my infosec friends, I made the decision that security was the field for me. I had just attended DEF CON 21, and gone to a BSidesSLC. I was enchanted! I had never felt like I met a family of people until I got to go wild with a bunch of hackers. I belonged. It was comfortable, fun, and inspiring like nothing else!
I recall attending a talk at DC21 and not even knowing what SSH was. Shortly after, I was thrust into my opportunity building servers and systems. I learned real quick drinking from a firehose, and being left to my own devices! And I was still thirsty.
It just so happens that with DC21, there was a resurgence in life for DC801. We had a (small) hackerspace (801 Labs), were meeting up regularly, and I got to engrain myself in the local community just a bit more. Eventually that space moved to its current location, where to this day I return regularly for motivation and experience!
I would not be where I am today without this hackerspace. Even though it's gone through its own transformation through the years, and its community has shifted, I relish every opportunity I’ve gotten from it.
One member got an amazing opportunity due to this community involvement. The offer was to come into a brand new security-focused startup, and architect out basically everything from scratch. He was only really known to the owners due to his outreach in the community, his reputation, appearing on panels, other cooperation with local security usergroups, etc.
At the end of 2015 I decided it was time to put serious effort into hopping over to security as a career. I had interviewed for a SOC intern position at a local company, and was given an offer for it. This would have been awesome! Except…
I got lucky. That member from the last section who got the opportunity to build out the company? He has been a friend of mine for years now, and of course, we met through DEF CON and the hackerspace. Said opportunity left him with the need to bring on someone to assist him with this monumental task. He sent me a message in IRC basically saying ‘DUDE ARE YOU READING CHAT?!?’’ because he had put out a general “Hey anyone looking for security work?” message in our channel, and thought I would have jumped all over it.
Of course the 1 day I’m not idling in IRC incessantly. Argh. Well, I give him a call. I got a phone interview with him and the bosses, with an offer to come down the next Friday and meet at a rent-a-boardroom type business park. 2 years later, we’ve built out a 5-man SOC and he’s a devoted security engineer/architect developing solutions. We grew the company over 10-fold in that time, and learned a WHOLE lot.
And here we are today. I consider myself a tier 2 security analyst worth his salt, getting to work directly with logs and PCAPs all day to find bad stuff going on. We’re helping to define a whole new arena of cybersecurity services (MDR - Managed Detection & Response), making a meaningful impact to help protect businesses of all shapes and sizes from security issues. I knew absolutely none of this before starting.
What wasn’t necessary was a grizzled security veteran (we had my buddy for that!), but someone that can take complex issues and break them down into bite-sized pieces that you can manage to. We started with log analysis, and quickly picked up network threat sensors for an additional data set. We’ve worked with partners to create custom SIEM solutions, and helped other companies mature their product offering.
We’re in an incredibly interesting time! There are absolutely core problems to IT security that do not have realistic market solutions. Network monitoring is a mess and complicated/expensive, no company is doing heuristics well, there’s a million different endpoint solutions with SIEM a constant nightmare due to the nature of logs.
But this is reality! This is what every organization faces, big or small. And there’s a place for smart, devoted people willing to tackle complex issues. It is the upcoming generation of security practitioners that will be finding valid solutions to a ton of issues. And that upcoming wave needs to be diverse, and capable at all degrees -- from people, to process, to technical.
At no time will you hear me upholding myself as smart -- the name Dumby wasn’t given to me by someone who understood the meaning of dumb (they were 2 years old), but it was oddly fitting. I don’t really know more about any 1 topic than my peers, but I’m able to take a little bit of everything and find helpful ways to apply those bits. Constantly evolving, and learning. I try to consume news of all sorts, curate my own social media feeds for technical data, and building up networks I can lean on when I need more expertise than my dumb self can provide.
And so far, it’s worked wonderfully! That gets us to today, where this story ends. One day I’ll find an intra-passion, uncovering a path to becoming SME on a particular technology or esoteric area of the industry, but this is not that time. I’ve continued down my path as a ‘Jack of all Trades’, and constantly amazed by the wide and terrible world of IT. There’s a small chance I might be getting pulled into the arena of data analytics, which is daunting and awesome!
I want to thank you for reading this piece. I hope it helped expose some pieces of the puzzle that may be missing in your own life. I am always available on Twitter to bullshit more, maybe say something useful, provide advice.
Follow me at https://twitter.com/uncl3dumby
I feel like I gotta do a victory dance for little old me.
From getting the skeleton of this website out, to finally really fleshing out the feature set... I feel things are finally coming along! I should only have very small things to improve on from here on out. I'm actually satisfied with how it looks, which is an insane feeling. From swearing at CSS while waiting on an oil change, to implementing stupid Quality-of-Life things like an in-browser editor for posts, doing stupid things like rm'ing my entire local development directory, this has been an adventure!
I'm still as big a noob as ever. However, I'm always amassing more knowledge! Hopefully I'll find ways for it to be useful!