My primary purpose for DumbyLand is to build up a sandbox for simulating different types of IT environments. A significant part of this is collecting logs for analysis to build up my blue team skills further. Architecting, engineering, and deploying SIEM and some basic controls are a huge part of that.
I want to be able to play around with different technologies, but to get things rolling, it’s best to just choose 1 thing at a time that sounds fun and interesting to set up. Graylog is something I’ve been hearing a lot about, it’s definitely something worth checking out and learning!
Instructions to get going for Graylog seem simple enough! Download the OVA, import it, start the VM, and wait.
It starts out with a simple setup of 4GB RAM, 20GB disk, and 2 CPU. I dig this, much preferred to SIEMonster and others that demanded 5 separate systems to get started, most requiring 8GB RAM each! Not something I can do until I set my server up with more RAM for a few hundred more bucks… which I’m hoping to hold off as long as possible to see if this price war lets up.
Enough of that, though. The system got a DHCP lease, and I hit it from the web interface.
Boom! Simple as that! Login with default admin/admin, and we’re rolling.
1. Send in first log messages
Let’s do it! We’ve got a basic Windows environment already going in the lab, so let’s pull it in.
From reading that community page, I understand that nxlog is going to be the best option for forwarding logs, and I’m always down to learn something new. Unfortunately, Graylog just leaves us with a “they’ll pick it up from here”.
Over on the nxlog website, there are downloads for the community edition for Windows machines. Documentation seems solid so I gave it a pass through reading about features and the basic functionality of it. This will be great to review later in-depth for a good understanding, but I think right now it’s most important to just get it up and going to see what we’ve got.
We’re going to go with a basic agent-based collection (Section 3.3.1) which should require something running nxlog on the Graylog server but we’ll get to that when we need to.
RDP in to my Windows desktop VM, and then to my DC.
Since I don't want to abandon all good security practice, I’m not gonna go around simply disabling these controls until I have good reason to. To allow this download, I had to add nxlog.co to my trusted sites. Search for ‘Internet Options’, then to Security tab, Trusted Sites, and then Sites to add the URL for the agent. Phew! But the rest was pretty straight forward. Downloaded the .msi, ran it, all good!
After the install finished I should have expected the readme to pop up, but it didn’t. Back to the documentation! The default installation location for me was “C:\Program Files (x86)\nxlog” which contains the conf folder.
In the documentation (Chapter 33. Microsoft Windows) there is a section for advice/recommendations/setup for Windows logging. For now, I really only care about Windows EventLog which there’s a jump to (Chapter 80. Windows Event Log). From there, I jump to (80.1. Local Collection with im_msvistalog) which is used for any system running 2008/Vista or later.
They provide an example of collecting all Windows EventLog data in JSON, written to a local file. I dunno exactly where we’ll go from here to get it to Graylog, but it’s at least the right direction! So let’s do that. Back on my DC, I’m gonna leave basically all of the conf alone, and just add the snippets that are included inthe docs here.
I forgot to open Notepad with admin privs, but also realized that even though the basic config file is super basic I should create a conf backup… good habits and all. Alright!
Now that we’ve got some basic configuration which we can revisit later, let’s get it on (12.1.1. Installing Interactively) and we’re up step 5. to verify the configuration, which passed. Now I opened services by searching ‘Services’, opened it, and then started the nxlog service.
The test config points the json output to “C:\test\sysmon.json” which I hadn’t created the dir for before starting the service. Maybe that’s an issue in getting it to write, but not sure. I double-checked config compared to what was in the example, and then also pulled up the logs for nxlog itself (C:\Program Files (x86)\nxlog\data\nxlog.txt) and saw that I’m getting a message of “WARNING not starting unused module eventlog” which makes me think that even though I defined an input, it doesn’t know what to do with the information.
Sure enough, there’s also an additional error message “WARNING no routes defined!”. I took a quick look at an example from Loggly for nxlog https://www.loggly.com/docs/logging-from-windows/ and found an example route that I used to copy and make one for myself. They have a section “internal” that I’m not sure what it does, but is an additional data source to investigate later.
define ROOT C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert
define CONFDIR %ROOT%\conf
define LOGDIR %ROOT%\data
define LOGFILE %LOGDIR%\nxlog.log
AutodetectCharsets iso8859-2, utf-8, utf-16, utf-32
# Check the size of our log file hourly, rotate if larger than 5MB
Every 1 hour
Exec if (file_exists('%LOGFILE%') and \
(file_size('%LOGFILE%') >= 5M)) \
# Rotate our log file every week on Sunday at midnight
Exec if file_exists('%LOGFILE%') file_cycle('%LOGFILE%', 8);
Exec $Message = to_json();
Path eventlog => file
This is my full conf for a test setup working. Hooray!! It’s very straightforward and simple. Eventlog module pulls down the information, and then uses the defined output ‘file’ to write it to a file all in the magic of json. We’ll see if I have any troubles but at least this is on the road to progress!
I need to configure my Graylog server with a static IP so that I can start to forward stuff to it, then I’ll configure nxlog to forward over the network. Back in the Graylog webconfig, I poked around the options, but doesn’t seem like it will be that easy, which is understandable.
They instruct you to go about it the standard way of editing /etc/network/interfaces and configuring there, as I’ll do now. Since you’re modifying the network connection, directly console in via ESXi and login with ubuntu/ubuntu, and you can follow the instructions below but it’s a standard configuration of the interface.
Logged back into the web config at 192.168.2.20, woohoo! This is all set. Now just to update the configuration on my DC to push towards this IP address.
Slight pivot on configuration. I’m having trouble with Graylog inputs. I get the general gist, but get these “failures” on trying to start the inputs. Not eactly sure what’s wrong, but going to attempt a new thing here.
This is a “content pack” which is a bunch of pre-configured stuff for Graylog, this one specifically for AD security auditing. It seems pretty damn nifty, and the Github includes a nxlog configuration to use. Import the content pack via Graylog webUI, and it will create the input and some dashboards and a couple other things.
Still having trouble getting this new input to work, I did what works best for anyone… I did a nice little restart on the Graylog host. Running ‘netstat -tulnp’ showed me that the designated port 5414 was running, but only on udp6. Perhaps this is just me being misinformed on how things should look, but I’m now the proud recipient of AD logs!
This is really getting my server to spin. It seems like Graylog is starting to choke up, so I’m gonna give it a minute since there was a sudden spike in activity overall and see if things normalize some.
And overall, my server is feeling like this with that
After giving it a while and still not getting a response when searching for messages in Graylog, I figure it’s time to look for more help, perhaps bumping up its CPU availability some.
I ended up not having to do this… yet, at least. I went back and reviewed the configuration steps, re-performed the basic graylog-ctl stuff, and then did the reconfigure command. I went back to the web UI, did a search against the stream from my Windows log input, and…
We have liftoff! Whoooooooooooooooooooooooooo!!! Now that we have a SIEM up and going, I can work piecemeal to have them ingested into Graylog as I bring services and additional machines online. This is so very exciting for me!
Next up I’d really like to be IDS, but I’m really not sure how I’m gonna get traffic mirroring set up on my box since I don’t have vCenter stuff rolling yet. I COULD just set that up, but that seems like a lot of effort for a single (important!) feature of this lab.
Also interestingly enough… now that I got everything working and am running queries against the data, my CPU usage dropped dramatically! It really seems like something got screwed up in the configuration process and the reconfiguration kicked everything back into gear.
For a minute I was really concerned that I was already making my box choke! WIth just 1 input into the SIEM and no heavy queries or anything, already maxing it at a sustained 90% leaving my box at just shy of 50% utilization had me scared. Glad to see things are REALLY starting to work out! Even at the most desperate of times, my server hit a maximum load of 57.4% and an average of 37.44% over the last hour where the vast majority of the time it was in that CPU % funk.
I think that’s pretty god damn impressive, if I may say so myself. To review, this is currently what we’ve got running:
Hypervisor Host - ESXi (Supermicro E300-8D)
Opnsense - Core Router
192.168.1.1 / 192.168.2.1
Windows 2016 - Domain Controller
Windows 7 - Client Machine / Admin Machine
Windows 7 - Victim Machine
And let’s keep going! Since things are working well right now, I grabbed a snapshot of Graylog as well as my domain controller. My progress so far has been quite pleasing!
Reviewing OPNsense configuration again real quick, I was reminded that it has built-in IDS features! Doing light digging, I discover it’s based on Suricata, cool!
Let’s try it out! To make the most use of the work I just completed, and to have a bit more fun, let’s also get OPNSense set up with sending its logs to Graylog. This is considered “remote logging” in OPNSense terminology and to be honest was a pain in the ass to find proper documentation for… I never found any.
To get started, navigate to “System > Settings > Logging”
Seems like it should be straight forward, but let’s see!
I have configured Graylog with the following content pack for PfSense since it should work just fine for OPNSense
Naturally… things aren’t straightforward. Now my Graylog isn’t getting Windows logs, what the fuck! Back to basics with troubleshooting this. Everything seems good with nxlog configuration, and the DC can ping my Graylog server. Given my inconsistency earlier, I’m gonna try to just restart Graylog and see what happens.
Back to inputs, pull back all messages for Windows, and sure enough… there they are! There’s definitely some inconsistency going on with my set up. I believe it’s likely Elastisearch since all the data going back over an hour is suddenly available in the UI to search for… bah! Oh well, the joys of simulating and labs!
Guess what else we’re getting now! That’s right…
Woot! That content pack has a Pfsense-Logs input with extractors for a few different things. It pulled out the authentication logs in opnsense with a good source, so I think that should definitely satisfy my basic needs!
This screen makes me happy. Very happy indeed! Quite satisfied with how it’s handled the load so far, even if some things have been a bit funky. Maybe ECC RAM is gonna be a help? I’ll spend a day on stability and load testing in the future once more things are running.
Now that I've got a SIEM and IDS features to play around with, I can start to bring on additional services and expand capability and scope! I have plans to bring on representations of development and production servers. In the meantime I also really want to bring up some vulnerable servers to play around with pentesting, maybe try out some brute forcing fun, throw up Mimikatz as well as some Icebreaker (https://github.com/DanMcInerney/icebreaker) and other MITM goodies. DNS poisoning, RAT, some general C2, data exfil... all that's good stuff I'll be getting to as I build this out!
|define ROOT C:\Program Files (x86)\nxlog|
# For windows vista/2008 and above use:
# For windows 2003 and earlier use the following:
# Module im_mseventlog
Path in => out