The Network Operations Center: Part 1

The day has finally come, it’s time to build our first official Network Operations Center (NOC) for my business. I never expected this day to come so early, but with proven results, the customer base has grown very quickly. I’m proud to say I’ve went from zero to over 95 sites protected of varying traffic levels with the total request volume being around 250,000 per day. I’ve already observed and blocked numerous denial of service attacks and hacking attempts.

In an accord to continually iterate and refine my offering, we now have a clean web interface coming down the line for all customers to see and manage their protected sites.

The current “NOC” is simply bash scripts and basic Prometheus exporters with email alerts on anomalies. It works, but it’s not clean and as visual as I’d like it. So, it’s time to build a proper NOC!

This posts goal is to outline what I’d like in the NOC, the security of the NOC, and what we’ll be monitoring! Without further ado, let’s get started!

What’ll be in the NOC?

I’ve been looking through some NOCs, and I’d like to monitor the network quality (incoming / outgoing, total consumption, origins, anomalies or “attacks”), the server statuses (load, CPU and steal time, # of requests per second, # of threats blocked, # of loaded rules, last sync status), and lastly I’d like to have a loaded list of who’s on call, ongoing tickets and server issues, and an easy way to access the NOC remotely while maintaining the security of the NOC.

NOC Security (Physical)

I’d like to make the NOC as secure as possible, from a physical and digital standpoint. Starting with our physical security first!

The NOC room will be a room with two large bay windows, they’ll have privacy shields on them, so there’s still light but it’s more difficult to see inside the NOC room.

I’d like to have RFID badges with proper controls on the doors, and full logging of who’s accessed, how long and when they were in the NOC.

Lastly, all systems in the NOC should be air-gapped, only be able to access the appropriate monitoring systems — any external network connectivity should go over a VPN so we can track it.

NOC Security (Virtual)

For the virtual aspect of the NOC, when anyone needs to connect remotely they’ll need to connect to the WireGuard VPN, sign-in to the NOC with multi-factor authentication. In terms of multi-factor authentication, I’d like Credentials + 2FA + Security Key. Individuals may only connect to the NOC on company supplied laptops.

Physical Displays

The NOC wall will contain 2 50” televisions connected via HDMI cords to one display PC, it’ll have Grafana open 24×7 and will have in one corner the on-call list for all regions.

Summary

It will be a lot of work to create this NOC monitoring room, and will take some engineering to implement the controls I desire, but in the end I believe it will be a secure, scalable NOC as the infrastructure grows and the team evolves.