pfSense Lab Part 3: IDS/IPS Setup with Snort

This lab series guides you from the initial deployment of pfSense. Then, it progresses to advanced configurations. It provides a structured, step-by-step journey through every major feature. If you missed any of the other parts, they are linked below. This way, you can catch up and get the most from the full lab experience.

As a reminder, here’s the architecture below we’re working with.

What are IDS and IPS? Comprehensive Overview of Intrusion Detection and Prevention Systems

Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) are critical components of modern network security infrastructures. IDS monitors network activity and system events to detect signs of possible security incidents, violations, or imminent threats against your security policies. Essentially, it acts as a vigilant observer that analyses traffic patterns and behaviors, alerting administrators upon detecting suspicious activity.

Prevention, on the other hand, takes the concept a step further. Intrusion Prevention Systems (IPS) not only detect attacks or policy violations, but also actively block or stop these incidents from causing harm. Together, IDS and IPS form a two-pronged defense: detection combined with response.

The Challenge of Modern Networks

Enterprise networks typically interface with multiple external public and private networks while needing to remain accessible to users and clients. The crux of maintaining such a network lies in balancing openness with security. However, attacks have evolved to be extremely sophisticated — often able to bypass traditional safeguards such as encryption and firewalls. These conventional technologies alone no longer suffice to protect against today’s complex threats, making IDS and IPS essential.

IDS/IPS Capabilities

Continuous Monitoring: IDS/IPS operate 24/7, analyzing network traffic and system logs for signs of trouble.
Alerting and Logging: IDS will log suspicious activities and generate alerts for administrator action. IPS complements this by blocking or mitigating threats immediately.
Policy Enforcement: Both systems can help enforce organizational security policies, such as preventing unauthorized VPN access.
Threat Intelligence: Through signature-based, anomaly-based, and protocol-state analysis, IDS/IPS detect both known and novel threats.
Automation: IPS provides real-time automated protection, reducing reliance on manual intervention.

Together, IDS and IPS markedly improve security by detecting and neutralizing threats within network boundaries before damage occurs.

How Do IDS and IPS Work?

Intrusion Detection Systems (IDS) generally use three primary detection methodologies to identify possible incidents:

Stateful Protocol Analysis:
This approach compares observed network protocol behavior against predetermined profiles of acceptable, benign activity for each protocol state. By detecting deviations from these accepted protocol states, stateful protocol analysis can discover subtle attacks that other techniques may miss.

Signature-Based Detection:
This method compares observed events—such as network packets or log entries—against a database of known threat signatures. It is the simplest detection type, operating by pattern-matching strings to find potential incidents that match known malicious signatures.

Anomaly-Based Detection:
Anomaly detection works by defining what is considered normal activity for your network. It then monitors real-time events to identify significant deviations from this baseline, effectively identifying previously unknown or novel threats that do not match existing signatures.

Why IDS and IPS Are Essential for Cybersecurity

Security teams today face the increasing threat of data breaches and regulatory fines while navigating budget constraints and organizational policies. IDS (Intrusion Detection Systems) and IPS (Intrusion Prevention Systems) address key aspects of a cybersecurity strategy by:

Automation: IDS/IPS operate largely autonomously, making them ideal components in modern security stacks. IPS, in particular, provides peace of mind by proactively protecting the network against known threats with minimal resource overhead and administrative intervention.
Compliance: Many compliance frameworks require organizations to invest in technologies like IDS/IPS to protect sensitive data. Implementing these systems helps meet CIS security controls and other regulatory requirements while providing valuable audit data for investigations and audits.
Policy Enforcement: IDS/IPS can be configured to enforce internal security policies at the network level. For instance, if only a single VPN should be permitted, IPS can block unauthorized VPN connections, ensuring adherence to organizational rules.

By detecting and mitigating attacks early, IDS and IPS reduce the risk of costly data breaches, downtime, and non-compliance penalties. They deliver critical visibility and control, enabling security teams to focus efforts on emerging threats and strategic defenses.

What is a Host-Based Intrusion Detection System (HIDS)?

A Host-Based Intrusion Detection System (HIDS) is a specialized security tool designed to monitor and analyze the internal components of a single computer or system. Unlike Network IDS (NIDS), which monitors traffic across the entire network, HIDS focuses on one host—such as a workstation, server, or endpoint—inspecting system activity including network packets on its interfaces as well as system files and processes.

HIDS represent the earliest form of intrusion detection software, originally designed for computers with limited external interaction, such as mainframes. Today, HIDS are highly configurable and capable of monitoring dynamic system behavior according to customized policies.

How Does HIDS Work?

Beyond analyzing targeted network packets, a HIDS monitors which programs access specific system resources, providing detailed insight into host behavior. Typically, it uses a database—often called an object database—containing monitored system objects such as critical files, directories, and processes.

For each monitored object, the HIDS records attributes like permissions, size, modification dates, and calculates checksums (e.g., MD5 or SHA1 hashes) for file contents. These data form a secure baseline stored in a checksum database. During operation, HIDS compares current system states against this baseline to detect integrity violations or unauthorized changes, signaling possible compromise.

Snort: A Powerful Open-Source IDS and IPS

Snort is a widely-used, powerful open-source Intrusion Detection System (IDS) and Intrusion Prevention System (IPS) that provides real-time network traffic analysis and packet logging. It functions by using a rule-based language that combines several inspection methods—signature-based, anomaly-based, and protocol analysis—to detect potentially malicious activity on a network.

With Snort, network administrators can detect a variety of sophisticated attacks such as Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS), Common Gateway Interface (CGI) attacks, buffer overflows, and stealth port scans. Snort achieves this by matching network packets to a comprehensive set of rules that define malicious behavior. When such a match is found, Snort generates alerts and can also take action to block the traffic in IPS mode.

Snort is free and open source, making it accessible for individuals and organizations of all sizes. It supports multiple operating modes:

Packet Sniffer Mode: Similar to tools like tcpdump or Wireshark, Snort captures and displays network packets in real-time.
Packet Logger Mode: Snort logs network packets to disk for analysis and archiving.
Network Intrusion Prevention System (NIPS) Mode: Operating inline, Snort can detect and actively prevent threats by blocking malicious packets.

The rules Snort uses come in two main sets:

The Community Ruleset, which is freely available and maintained by the community.
The Snort Subscriber Ruleset, which is developed and maintained by Cisco Talos and delivered in real-time to subscribers.

Snort’s powerful rule language allows detailed customization on what traffic to monitor and how to react, making it an invaluable tool for protecting networks against a wide range of cyber threats.

Key Features of Snort IDS/IPS

Snort is a powerful open-source Intrusion Detection System (IDS) and Intrusion Prevention System (IPS) that provides real-time network traffic analysis and packet logging. Its effectiveness comes from a rich set of features that enable network administrators to monitor and detect malicious activity efficiently:

Real-Time Traffic Monitoring:
Snort monitors incoming and outgoing traffic on IP networks. It inspects packets in real time and sends alerts when it detects potentially malicious or suspicious activity, providing immediate awareness to administrators.
Packet Logging:
In its packet logger mode, Snort records all packets to disk. These records are organized hierarchically based on the IP address of the host network, facilitating detailed forensic analysis and troubleshooting.
Protocol Analysis:
Snort captures data across various protocol layers for deep inspection. This allows administrators to analyze potentially harmful packets with precision, especially for protocols within the TCP/IP stack.
Content Matching:
Snort’s rules are categorized by protocol (such as IP and TCP), and further by ports and content presence. It uses a multi-pattern content matching engine that improves performance, especially for protocols like HTTP. Rules without content checks are still evaluated but can impact performance negatively.
Rule-Based Language:
Snort’s flexible rule language enables administrators to define what network traffic to collect and specify actions to take on detecting malicious packets. This helps distinguish between normal and malicious network activities.
Multi-Mode Operation:
Snort supports various modes including packet sniffer, packet logger, and inline IDS/IPS mode where it can actively block malicious traffic

The Different Operation Modes of Snort

Snort can be configured to operate in three primary modes, each serving a specific purpose depending on your network monitoring needs:

Packet Sniffer Mode:
In this mode, Snort reads network packets and displays them continuously on the console. This mode functions similarly to a packet capture tool like tcpdump, allowing network administrators to monitor real-time traffic flow on their networks.
Packet Logger Mode:
When operating as a packet logger, Snort records all network packets it observes onto disk. These saved packets can be reviewed later for forensic analysis or troubleshooting. The logged data is typically organized in directories by host IP address, helping systematically store captured traffic.
Network Intrusion Prevention Detection System (NIPDS) Mode:
This is Snort’s intrusion detection and prevention mode. In NIPDS mode, Snort actively monitors and analyzes network traffic against a set of user-defined rules that specify malicious patterns or behaviors. When such threats are identified, Snort can generate alerts and also take preventive actions—such as blocking malicious packets—to protect the network. The specific actions Snort takes depend on its rule configurations, allowing for tailored security responses.

Implementing Snort on Our Network Topology

Fortunately, the pfSense project simplifies the deployment of Snort by offering it as an easily installable package through its Package Manager. This integration makes managing, updating, and maintaining Snort straightforward, even for those new to IDS/IPS deployment.

To install Snort on pfSense, simply navigate to the Package Manager, browse the available packages, and select Snort. With just a few clicks, Snort is installed and ready for configuration within your firewall environment.

Now that Snort is installed on pfSense, navigate to Services > Snort within the web interface. The first important step is to select the network interface you want Snort to listen on and monitor.

Choosing the correct interface to sniff traffic from is critical because Snort analyzes all packets passing through that interface for potential malicious activity. For example, you might select the WAN interface to monitor incoming external traffic or a LAN interface to monitor internal traffic

Within the Snort interface settings on pfSense, you first select the network interface that Snort will listen to and monitor for suspicious activity. For the IPS functionality, you can choose the legacy mode, which uses the pcap library to create a copy (or clone) of each packet as it arrives from the network card before it is processed by the pfSense firewall engine (pf).

This mode allows Snort to inspect all traffic without directly interfering with the flow, enabling it to detect and optionally block malicious activity effectively. Additionally, you have the option to enable logging, which allows Snort to record detected events for later analysis and auditing, enhancing network security monitoring capabilities.

Snort Rule Sets and Their Usage

Snort operates using detection signatures called rules. These rules define patterns of malicious network behavior and dictate what traffic should be monitored and how to respond when suspicious activity is detected. Users have the flexibility to create custom Snort rules, or they can enable and download from a variety of pre-packaged rule sets.

The Snort package on pfSense supports several important pre-packaged rule sets:

Snort VRT (Vulnerability Research Team) Rules:
Developed, tested, and approved by Cisco Talos Security Intelligence and Research Group, these rules are part of the subscriber rule set. The subscribed version offers the latest threat detection rules, updated frequently to address evolving threats. A free registered-user version is also available but with a delay in updates.
Snort GPLv2 Community Rules:
This is a subset of the subscriber rules, released under GPLv2 license and available freely without registration. It is updated daily and provides solid baseline protection for users who do not want to subscribe.
Emerging Threats Open Rules:
An open-source, community-driven rule set designed to improve malware detection capabilities. It’s ideal for users with budget constraints who want enhanced IDS/IPS anti-malware coverage.
OpenAppID:
Contains open detectors and rules for application identification used by Snort’s AppID preprocessor, allowing deeper application-layer visibility and filtering.

Selecting and regularly updating rule sets is critical to maintaining effective network defense, as new vulnerabilities and attack techniques continuously emerge.

The Updates tab in the Snort package interface is critical for maintaining the effectiveness of your Intrusion Detection and Prevention System. This tab provides an overview of the status of downloaded rule packages and enables you to retrieve new updates seamlessly.

A table within the interface displays the available Snort rule packages along with their current state—whether they are not enabled, not yet downloaded, or have valid checksums (MD5) with their respective timestamps. This allows administrators to quickly verify rule freshness and integrity.

When you click the Update Rules button, pfSense will connect to the respective vendor websites to download the latest rule sets. The system compares the MD5 checksum of the locally stored rule files with the remote files. If there’s a difference, indicating new or altered rules, the new files are downloaded and installed to keep your detection capabilities up-to-date.

Additionally, a FORCE button lets you bypass checksum verification and re-download the rule packages, useful for troubleshooting or ensuring the latest rules are in use.

Once you have updated the Snort rule sets, the next step is to run Snort. Running Snort activates the IDS/IPS engine with the latest detection rules loaded.

In any IDS/IPS system, the occurrence of false positives—alerts triggered by benign activity mistaken for malicious behavior—is inevitable. Snort on pfSense provides a powerful feature called Suppression Lists, which allow you to manage and reduce false alerts effectively.

When a false positive is identified, instead of disabling the entire rule (which could lead to missed detections elsewhere), you can add the specific alert or IP address to a suppression list. This instructs Snort to ignore that particular alert during rule checks.

Conclusion:

This blog post comprehensively covers the implementation of Intrusion Detection and Prevention Systems (IDS/IPS) using Snort on pfSense, detailing the critical role these systems play in modern network security. It has outlined how pfSense simplifies the deployment and management of Snort through its package manager, allowing seamless integration, frequent updates, and ease of maintenance.

The explanation included Snort’s key functionalities such as real-time traffic monitoring, packet logging, and protocol analysis, alongside its flexible multi-mode operation for sniffing, logging, and active prevention. The discussion on Snort’s rule sets highlighted the importance of using curated and regularly updated detection signatures to maintain strong protection against emerging threats. Furthermore, we explored practical operational aspects, including selecting interfaces, choosing IPS modes, updating rules, and handling false positives using suppression lists.

Altogether, this equips network administrators and cybersecurity professionals with the knowledge to effectively deploy Snort on pfSense, creating a dynamic and resilient defense layer that constantly protects against sophisticated cyber threats while maintaining operational efficiency.

Amatoch