Suricata Suricata - IDS on AsusWRT Merlin

[glehel]

AX88U has improved the processing speed of the thread auto I set. This means:
- Going to use 4 thread (s)
- all 8 packet processing threads, 2 management threads initialized, engine started.

increased memory usage but still have about 100 Mb free
I couldn't check the blocking yet to make it really work.

 

[Mutzli]

i am back to IDS mode as for me after 1 day in IPS it started dropping near any communication - DNS problems, even the router page was blocked. SSH was actively blocked too for some time:
f.e. it connects but then after one min it dropped connection actively and don't allow connect again, but after some retries, it connects for some time

using juched script + YAML with enabled IPS from faux123.

I have the same issues, but only on one of the Suricata installations. I'm running two different networks, both with the same router and as far as I can tell they run the exact same configuration. Only one of the networks has issues of dropping clients and connections. The other one runs with no problems. The Suricata installation without the issues is the one that has more clients and much more requests to handle. I haven't given up yet on IPS, I'm still trying to isolate what is the trigger that is causing Suricata to lock out clients.

 

[Milan]

I have the same issues, but only on one of the Suricata installations. I'm running two different networks, both with the same router and as far as I can tell they run the exact same configuration. Only one of the networks has issues of dropping clients and connections. The other one runs with no problems. The Suricata installation without the issues is the one that has more clients and much more requests to handle. I haven't given up yet on IPS, I'm still trying to isolate what is the trigger that is causing Suricata to lock out clients.

so I am not alone. I am not giving too, trying to isolate the reason ...

when I enable IPS mode then immediately all android devices are having connection problems - this I just observed. f.e. native youtube and Netflix apps are not able to connect and when they connect then only for short time and they get disconnected ...

 

[XIII]

For the people that seem to have IPS working without the fork:

What’s the result of blocking the IP of a website and then trying to access that website?

(browser, wget, curl)

 

[Mutzli]

so I am not alone. I am not giving too, trying to isolate the reason ...

when I enable IPS mode then immediately all android devices are having connection problems - this I just observed. f.e. native youtube and Netflix apps are not able to connect and when they connect then only for short time and they get disconnected ...

I completely removed Suricata this morning from the one router which had problems and reinstalled it from juched's github source as on page one of this thread. After the install I configured the .yaml with the following parameters and it is still working after 3h:

%YAML 1.1
---

# Holds variables that would be used by the engine.
vars:

  # Holds the address group vars that would be passed in a Signature.
  address-groups:
    HOME_NET: "[192.168.1.0/24]"
    EXTERNAL_NET: "any"
    DNS_SERVERS: "[192.168.1.1]"
    SMTP_SERVERS: "$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    DNP3_SERVER: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_SERVER: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_SERVER: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
    FTP_SERVERS: "$HOME_NET"
    SSH_SERVERS: "$HOME_NET"
    
 # Holds the port group vars that would be passed in a Signature.
  port-groups:
    FTP_PORTS: "21"
    HTTP_PORTS: "80"
    ORACLE_PORTS: "1521"
    SSH_PORTS: "29100"
    SHELLCODE_PORTS: "!80"
    DNP3_PORTS: "20000"
    FILE_DATA_PORTS: "$HTTP_PORTS,110,143"   
 
# Runmode the engine should use.
runmode: workers

# If set to auto, the variable is internally switched to 'router' in IPS
# mode and 'sniffer-only' in IDS mode.
host-mode: auto

# Linux high speed capture support
af-packet:
  - interface: eth0
    threads: 2
    defrag: no
    cluster-type: cluster_flow
    cluster-id: 99
    copy-mode: ips
    copy-iface: br0
    buffer-size: 3072
    use-mmap: yes
    tpacket-v3: yes
    ring-size: 3072
  - interface: br0
    threads: 2
    cluster-id: 98
    defrag: no
    cluster-type: cluster_flow
    copy-mode: ips
    copy-iface: eth0
    buffer-size: 3072
    use-mmap: yes
    tpacket-v3: yes
    ring-size: 3072
#  - interface: wl0.2

# IPS Mode Configuration
# PCAP
pcap:
  - interface: eth0
    checksum-checks: auto
    promisc: yes
  - interface: br0
    checksum-checks: auto
    promisc: yes

pid-file: /opt/var/run/suricata.pid

legacy:
  uricontent: enabled

max-pending-packets: 1024

Fingers crossed and hopefully this is fixing my issues. I'll let you know if it blocks local clients again.

 

[crkpot]

i am back to IDS mode as for me after 1 day in IPS it started dropping near any communication - DNS problems, even the router page was blocked. SSH was actively blocked too for some time:
f.e. it connects but then after one min it dropped connection actively and don't allow connect again, but after some retries, it connects for some time

using juched script + YAML with enabled IPS from faux123.

I ran into the exact same issues you describe. I'm also behind an AT&T gateway that only supports ip passthrough mode which has been fine, but after a day or so my AX58U lost the public ip from the gateway and couldn't renew it. My server pc also lost it's assigned ip from the AX58U and dhcd'd a random new one. Speed tests and Samba were maxing maxing the cpu at half the normal speeds. I didn't have time to investigate or tinker with configs so I just uninstalled suricata for now.

 

[rgnldo]

I tested here all environments and routing, including the one with the manageable switch, running smoothly. Interesting to watch the reports. I cannot understand the Suricata's participation in these Problems. Does not make sense. They will soon find the solution.

 

[Milan]

ok one reason for blocking ssh when in IPS mode:

suricata[10037]: [1:2003068:7] ET SCAN Potential SSH Scan OUTBOUND

how to disable it ?

 

[Mutzli]

ok one reason for blocking ssh when in IPS mode:

suricata[10037]: [1:2003068:7] ET SCAN Potential SSH Scan OUTBOUND

how to disable it ?

What do you have set under SSH_PORTS in the .yaml? Does it match your SSH port configured to the router?

 # Holds the port group vars that would be passed in a Signature.
  port-groups:
    FTP_PORTS: "21"
    HTTP_PORTS: "80"
    ORACLE_PORTS: "1521"
    SSH_PORTS: "22"

 

[Milan]

What do you have set under SSH_PORTS in the .yaml? Does it match your SSH port configured to the router?

 # Holds the port group vars that would be passed in a Signature.
  port-groups:
    FTP_PORTS: "21"
    HTTP_PORTS: "80"
    ORACLE_PORTS: "1521"
    SSH_PORTS: "22"

ahh fixed. will see, another try now ...

 

[Mutzli]

Fingers crossed and hopefully this is fixing my issues. I'll let you know if it blocks local clients again.

Sad to report that it locked me out again this morning. No clue given in fast.log about what was blocked. Back to IDS only again. It is impossible to investigate if Suricata doesn't log what ports are being blocked and why.

 

[faux123]

I believe I have cracked the egg on the right settings for our router. I will let it brew for a while before I share it (think of this as internal beta testing).

I was having some bad issues such as high TCP reassembly, kernel packet drops, intermittent connection issues etc etc, now it's fairly stable and inline IPS mode is working as expected. Took hours and hours of wireshark debugging and log reading to isolate issues (BTW, any connection issues, wireshark is the way to go)!

One major side issue I've also discovered and wanted to share now, if you have any DUAL-STACK (IPV4/IPV6) enabled on our router, Suricata 4.1.8 version will have connection issues. Either stick to IPV4 only or IPV6 only but NOT both regardless of method (tunnel, 6in4, etc, etc). I'm disappointed with this discovery but I will trade my IPV6 tunneling for a working IPS at this moment....

 

[heysoundude]

I believe I have cracked the egg on the right settings for our router. I will let it brew for a while before I share it (think of this as internal beta testing).

I was having some bad issues such as high TCP reassembly, kernel packet drops, intermittent connection issues etc etc, now it's fairly stable and inline IPS mode is working as expected. Took hours and hours of wireshark debugging and log reading to isolate issues (BTW, any connection issues, wireshark is the way to go)!

One major side issue I've also discovered and wanted to share now, if you have any DUAL-STACK (IPV4/IPV6) enabled on our router, Suricata 4.1.8 version will have connection issues. Either stick to IPV4 only or IPV6 only but NOT both regardless of method (tunnel, 6in4, etc, etc). I'm disappointed with this discovery but I will trade my IPV6 tunneling for a working IPS at this moment....

does the entirety of this pertain to your fork specifically, or both Merlin and your fork? If the former, may I humbly request you begin your own topic/thread so as not to confuse the rest of us please?

 

[rgnldo]

I understand the need for the organization, it is legitimate. But the post is related to the thread, since it is Suricata, via entware. Any link to the fw fork will be mentioned.

 

[rgnldo]

wireshark

hard work

Sniffing Packets with Wireshark

Suriwire is a plugin for wireshark that allow you to display suricata

 

[faux123]

does the entirety of this pertain to your fork specifically, or both Merlin and your fork? If the former, may I humbly request you begin your own topic/thread so as not to confuse the rest of us please?

No, the settings should be independent of firmware base. With the setting I'm running at this moment, it is very memory intensive as Suricata is a memory hog to run in IPS mode.

 

[juched]

No, the settings should be independent of firmware base. With the setting I'm running at this moment, it is very memory intensive as Suricata is a memory hog to run in IPS mode.

Good to know. The issues of stopping connections etc after a bit of running does sound like memory exhaustion. Have been too busy to play with IPS mode, but was looking to come back to this. Looking forward to what you find. It is appreciated.

 

[heysoundude]

No, the settings should be independent of firmware base. With the setting I'm running at this moment, it is very memory intensive as Suricata is a memory hog to run in IPS mode.

ok - doesn't the swap most of us install help? I'd have to check, but I seem to recall I've got 2GB set up for that that I've only seen get gently caressed when I've looked - does IPS look for more than that?

 

[rgnldo]

I believe that IPS mode via af_packet on FW is not possible.
via pcap, yes. No errors.
I think it has to do with the NIC driver´s compatibility
Viable and lightweight IPS mode is via NFQ netfilter, with iptables, in linux.
I removed rules that will give problems for less experienced users.

for tests:
run and verify. After return, ctrl +C.

suricata -c /opt/etc/suricata/suricata.yaml --af-packet --simulate-ips

%YAML 1.1
---

# Holds variables that would be used by the engine.
vars:

  # Holds the address group vars that would be passed in a Signature.
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
    EXTERNAL_NET: "any"
    SMTP_SERVERS: "$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    DNP3_SERVER: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_SERVER: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_SERVER: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
    FTP_SERVERS: "$HOME_NET"
    SSH_SERVERS: "$HOME_NET"
 
# Holds the port group vars that would be passed in a Signature.
  port-groups:
    FTP_PORTS: "21"
    HTTP_PORTS: "80"
    ORACLE_PORTS: "1521"
    SSH_PORTS: "29100"
    SHELLCODE_PORTS: "!80"
    DNP3_PORTS: "20000"
    FILE_DATA_PORTS: "$HTTP_PORTS,110,143"

# Runmode the engine should use.
runmode: workers

# If set to auto, the variable is internally switched to 'router' in IPS
# mode and 'sniffer-only' in IDS mode.
host-mode: auto

# Linux high speed capture support
af-packet:
  - interface: -> WAN
#    threads: auto
#    defrag: no
#    cluster-type: cluster_flow
#    cluster-id: 98
#    copy-mode: ips
#    copy-iface: br0
#    tpacket-v3: no
#    ring-size: 2048
#    use-mmap: yes
  - interface: br0
#    threads: auto
#    defrag: no
#    cluster-type: cluster_flow
#    cluster-id: 97
#    copy-mode: ips
#    copy-iface: ppp0
#    tpacket-v3: no
#    ring-size: 2048
#    use-mmap: yes

# - interface: wl0.2

# IPS Mode Configuration
# PCAP
pcap:
- interface: auto
   checksum-checks: auto
   promisc: yes
   copy-mode: ips
   copy-iface: br0

- interface: br0
   checksum-checks: auto
   promisc: yes
   copy-mode: ips
   copy-iface: ppp0
# - interface: wl0.2
#   checksum-checks: auto
#   promisc: yes

pid-file: /opt/var/run/suricata.pid

#legacy:
#  uricontent: enabled

# default-packet-size: 1480
# max-pending-packets: 1024

# The default logging directory.
default-log-dir: /opt/var/log/suricata

stats:
  enabled: no
  # The interval field (in seconds) controls at what interval
  # the loggers are invoked.
  interval: 8
  decoder-events-prefix: "decoder.event"

# Configure the type of alert (and other) logging.
outputs:

# a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: fast.log
      append: yes
      filetype: regular

  # alert output for use with Barnyard2
  - unified2-alert:
      enabled: no
      filename: unified2.alert
      limit: 32mb
      sensor-id: 0
      xff:
        enabled: no

  - http-log:
      enabled: yes
      filename: http.log
      append: yes
      extended: yes
      filetype: regular

  - pcap-log:
      enabled: no
      filename: log.pcap
      limit: 32mb
      max-files: 1000
      mode: normal

  - tls-log:
      enabled: no
      filename: tls.log
      extended: yes

  - tls-store:
      enabled: no
      certs-log-dir: certs

  - stats:
      enabled: yes
      filename: stats.log
      interval: 10
      append: no

  - syslog:
      enabled: no
      identity: suricata
      facility: local1
      level: notice

  - drop:
      enabled: yes
      filename: drop.log
      append: yes
      filetype: regular

  - file-store:
      enabled: no
      log-dir: files
      force-magic: no

  - file-log:
      enabled: no
      filename: files-json.log
      append: yes
      filetype: regular
      force-magic: no
      #force-hash: [md5]

  - dns-log:
      enabled: no
      filename: dns.log
      append: yes
      filetype: regular

  - eve-log:
      enabled: no
      filetype: regular
      filename: eve-%Y-%m-%d-%H:%M.json

      types:
        - alert:
          tagged-packets: yes
          app-layer: true
          flow: true
          rule: true
          metadata: true
          raw: false

        - drop:
          alerts: yes
          flows: all

        # - http
        # - dns
        # - tls

# Magic file. The extension .mgc is added to the value here.
magic-file: /opt/share/misc/magic

detect-engine:
  - profile: medium
  - custom-values:
      toclient-src-groups: 2
      toclient-dst-groups: 2
      toclient-sp-groups: 2
      toclient-dp-groups: 3
      toserver-src-groups: 2
      toserver-dst-groups: 4
      toserver-sp-groups: 2
      toserver-dp-groups: 25
  - sgh-mpm-context: auto
  - inspection-recursion-limit: 3000


# Suricata is multi-threaded. Here the threading can be influenced.
threading:
  set-cpu-affinity: no
  detect-thread-ratio: 1.0

# Defrag settings:
defrag:
  memcap: 32mb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 60

# Flow settings:
flow:
  memcap: 32mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  prune-flows: 5

# This option controls the use of vlan ids in the flow (and defrag)
# hashing.
vlan:
  use-for-tracking: true

# Specific timeouts for flows.
flow-timeouts:

  default:
    new: 30
    established: 300
    closed: 0
    emergency-new: 10
    emergency-established: 100
    emergency-closed: 0
  tcp:
    new: 60
    established: 3600
    closed: 120
    emergency-new: 10
    emergency-established: 300
    emergency-closed: 20
  udp:
    new: 30
    established: 300
    emergency-new: 10
    emergency-established: 100
  icmp:
    new: 30
    established: 300
    emergency-new: 10
    emergency-established: 100

stream:
  memcap: 32mb
  checksum-validation: yes      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    #raw: yes
    #segment-prealloc: 2048
    #check-overlap-different-data: true

# Host table is used by tagging and per host thresholding subsystems.
host:
  hash-size: 4096
  prealloc: 1000
  memcap: 16554432

# Host specific policies for defragmentation and TCP stream reassembly.
host-os-policy:
  # Make the default policy windows.
  windows: [0.0.0.0/0]
  bsd: []
  bsd-right: []
  old-linux: []
  linux: []
  old-solaris: []
  solaris: []
  hpux10: []
  hpux11: []
  irix: []
  macos: []
  vista: []
  windows2k3: []

# Logging configuration.  This is not about logging IDS alerts, but
# IDS output about what its doing, errors, etc.
logging:

  # This value is overriden by the SC_LOG_LEVEL env var.
  default-log-level: info

  # Define your logging outputs.
  outputs:
  - console:
      enabled: no
  - file:
      enabled: no
      filename: /opt/var/log/suricata/suricata.log
  - syslog:
      enabled: no
      facility: off
   
default-rule-path: /opt/var/lib/suricata/rules

rule-files:
   - botcc.rules
   - botcc.portgrouped.rules
   - compromised.rules
   - drop.rules
   - dshield.rules
   - emerging-malware.rules
   - emerging-mobile_malware.rules
   - emerging-worm.rules
   - ciarmy.rules
   - emerging-attack_response.rules

classification-file: /opt/etc/suricata/classification.config
reference-config-file: /opt/etc/suricata/reference.config
threshold-file: /opt/etc/suricata/threshold.config

# action-order:
#   - pass
#   - drop
#   - reject
#   - alert

# Limit for the maximum number of asn1 frames to decode (default 256)
asn1-max-frames: 256

engine-analysis:
  rules-fast-pattern: yes
  rules: yes

#recursion and match limits for PCRE where supported
pcre:
  match-limit: 3500
  match-limit-recursion: 1500

# Holds details on the app-layer. The protocols section details each protocol.
app-layer:
  protocols:
    krb5:
      enabled: no # Requires rust
    ikev2:
      enabled: yes
    tls:
      enabled: yes
      detection-ports:
      dp: "[443,444,465,853,993,995]"

    dcerpc:
      enabled: yes
    ftp:
      enabled: no
    ssh:
      enabled: yes
    smtp:
      enabled: yes
    imap:
      enabled: detection-only
    msn:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
          dp: 139, 445
    dns:
      global-memcap: 16777216
      state-memcap: 524288
      request-flood: 500

      tcp:
       enabled: yes
       detection-ports:
          dp: 53
      udp:
       enabled: yes
       detection-ports:
         dp: 53
    http:
       enabled: yes
       memcap: 16108864

###########################################################################
# Configure libhtp.
libhtp:
    default-config:
      personality: IDS
      request-body-limit: 2mb
      response-body-limit: 2mb
      request-body-minimal-inspect-size: 32kb
      request-body-inspect-window: 4kb
      response-body-minimal-inspect-size: 32kb
      response-body-inspect-window: 4kb
      http-body-inline: auto
      double-decode-path: no
      double-decode-query: no
    ntp:
      enabled: yes
    dhcp:
      enabled: yes
    sip:
      enabled: yes

coredump:
    max-dump: unlimited

# Suricata user pass through configuration

 

[juched]

I believe that IPS mode via af_packet on FW is not possible.
via pcap, yes. No errors.
I think it has to do with the NIC driver´s compatibility
Viable and lightweight IPS mode is via NFQ netfilter, with iptables, in linux.
I removed rules that will give problems for less experienced users.

for tests:
run and verify. After return, ctrl +C.

suricata -c /opt/etc/suricata/suricata.yaml --af-packet --simulate-ips

%YAML 1.1
---

# Holds variables that would be used by the engine.
vars:

  # Holds the address group vars that would be passed in a Signature.
  address-groups:
    HOME_NET: "[192.168.0.0/16,10.0.0.0/8,172.16.0.0/12]"
    EXTERNAL_NET: "any"
    SMTP_SERVERS: "$HOME_NET"
    HTTP_SERVERS: "$HOME_NET"
    SQL_SERVERS: "$HOME_NET"
    TELNET_SERVERS: "$HOME_NET"
    DNP3_SERVER: "$HOME_NET"
    DNP3_CLIENT: "$HOME_NET"
    MODBUS_SERVER: "$HOME_NET"
    MODBUS_CLIENT: "$HOME_NET"
    ENIP_SERVER: "$HOME_NET"
    ENIP_CLIENT: "$HOME_NET"
    FTP_SERVERS: "$HOME_NET"
    SSH_SERVERS: "$HOME_NET"

# Holds the port group vars that would be passed in a Signature.
  port-groups:
    FTP_PORTS: "21"
    HTTP_PORTS: "80"
    ORACLE_PORTS: "1521"
    SSH_PORTS: "29100"
    SHELLCODE_PORTS: "!80"
    DNP3_PORTS: "20000"
    FILE_DATA_PORTS: "$HTTP_PORTS,110,143"

# Runmode the engine should use.
runmode: workers

# If set to auto, the variable is internally switched to 'router' in IPS
# mode and 'sniffer-only' in IDS mode.
host-mode: auto

# Linux high speed capture support
af-packet:
  - interface: -> WAN
#    threads: auto
#    defrag: no
#    cluster-type: cluster_flow
#    cluster-id: 98
#    copy-mode: ips
#    copy-iface: br0
#    tpacket-v3: no
#    ring-size: 2048
#    use-mmap: yes
  - interface: br0
#    threads: auto
#    defrag: no
#    cluster-type: cluster_flow
#    cluster-id: 97
#    copy-mode: ips
#    copy-iface: ppp0
#    tpacket-v3: no
#    ring-size: 2048
#    use-mmap: yes

# - interface: wl0.2

# IPS Mode Configuration
# PCAP
pcap:
- interface: auto
   checksum-checks: auto
   promisc: yes
   copy-mode: ips
   copy-iface: br0

- interface: br0
   checksum-checks: auto
   promisc: yes
   copy-mode: ips
   copy-iface: ppp0
# - interface: wl0.2
#   checksum-checks: auto
#   promisc: yes

pid-file: /opt/var/run/suricata.pid

#legacy:
#  uricontent: enabled

# default-packet-size: 1480
# max-pending-packets: 1024

# The default logging directory.
default-log-dir: /opt/var/log/suricata

stats:
  enabled: no
  # The interval field (in seconds) controls at what interval
  # the loggers are invoked.
  interval: 8
  decoder-events-prefix: "decoder.event"

# Configure the type of alert (and other) logging.
outputs:

# a line based alerts log similar to Snort's fast.log
  - fast:
      enabled: yes
      filename: fast.log
      append: yes
      filetype: regular

  # alert output for use with Barnyard2
  - unified2-alert:
      enabled: no
      filename: unified2.alert
      limit: 32mb
      sensor-id: 0
      xff:
        enabled: no

  - http-log:
      enabled: yes
      filename: http.log
      append: yes
      extended: yes
      filetype: regular

  - pcap-log:
      enabled: no
      filename: log.pcap
      limit: 32mb
      max-files: 1000
      mode: normal

  - tls-log:
      enabled: no
      filename: tls.log
      extended: yes

  - tls-store:
      enabled: no
      certs-log-dir: certs

  - stats:
      enabled: yes
      filename: stats.log
      interval: 10
      append: no

  - syslog:
      enabled: no
      identity: suricata
      facility: local1
      level: notice

  - drop:
      enabled: yes
      filename: drop.log
      append: yes
      filetype: regular

  - file-store:
      enabled: no
      log-dir: files
      force-magic: no

  - file-log:
      enabled: no
      filename: files-json.log
      append: yes
      filetype: regular
      force-magic: no
      #force-hash: [md5]

  - dns-log:
      enabled: no
      filename: dns.log
      append: yes
      filetype: regular

  - eve-log:
      enabled: no
      filetype: regular
      filename: eve-%Y-%m-%d-%H:%M.json

      types:
        - alert:
          tagged-packets: yes
          app-layer: true
          flow: true
          rule: true
          metadata: true
          raw: false

        - drop:
          alerts: yes
          flows: all

        # - http
        # - dns
        # - tls

# Magic file. The extension .mgc is added to the value here.
magic-file: /opt/share/misc/magic

detect-engine:
  - profile: medium
  - custom-values:
      toclient-src-groups: 2
      toclient-dst-groups: 2
      toclient-sp-groups: 2
      toclient-dp-groups: 3
      toserver-src-groups: 2
      toserver-dst-groups: 4
      toserver-sp-groups: 2
      toserver-dp-groups: 25
  - sgh-mpm-context: auto
  - inspection-recursion-limit: 3000


# Suricata is multi-threaded. Here the threading can be influenced.
threading:
  set-cpu-affinity: no
  detect-thread-ratio: 1.0

# Defrag settings:
defrag:
  memcap: 32mb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 60

# Flow settings:
flow:
  memcap: 32mb
  hash-size: 65536
  prealloc: 10000
  emergency-recovery: 30
  prune-flows: 5

# This option controls the use of vlan ids in the flow (and defrag)
# hashing.
vlan:
  use-for-tracking: true

# Specific timeouts for flows.
flow-timeouts:

  default:
    new: 30
    established: 300
    closed: 0
    emergency-new: 10
    emergency-established: 100
    emergency-closed: 0
  tcp:
    new: 60
    established: 3600
    closed: 120
    emergency-new: 10
    emergency-established: 300
    emergency-closed: 20
  udp:
    new: 30
    established: 300
    emergency-new: 10
    emergency-established: 100
  icmp:
    new: 30
    established: 300
    emergency-new: 10
    emergency-established: 100

stream:
  memcap: 32mb
  checksum-validation: yes      # reject wrong csums
  inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
  reassembly:
    memcap: 64mb
    depth: 1mb                  # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
    #randomize-chunk-range: 10
    #raw: yes
    #segment-prealloc: 2048
    #check-overlap-different-data: true

# Host table is used by tagging and per host thresholding subsystems.
host:
  hash-size: 4096
  prealloc: 1000
  memcap: 16554432

# Host specific policies for defragmentation and TCP stream reassembly.
host-os-policy:
  # Make the default policy windows.
  windows: [0.0.0.0/0]
  bsd: []
  bsd-right: []
  old-linux: []
  linux: []
  old-solaris: []
  solaris: []
  hpux10: []
  hpux11: []
  irix: []
  macos: []
  vista: []
  windows2k3: []

# Logging configuration.  This is not about logging IDS alerts, but
# IDS output about what its doing, errors, etc.
logging:

  # This value is overriden by the SC_LOG_LEVEL env var.
  default-log-level: info

  # Define your logging outputs.
  outputs:
  - console:
      enabled: no
  - file:
      enabled: no
      filename: /opt/var/log/suricata/suricata.log
  - syslog:
      enabled: no
      facility: off
  
default-rule-path: /opt/var/lib/suricata/rules

rule-files:
   - botcc.rules
   - botcc.portgrouped.rules
   - compromised.rules
   - drop.rules
   - dshield.rules
   - emerging-malware.rules
   - emerging-mobile_malware.rules
   - emerging-worm.rules
   - ciarmy.rules
   - emerging-attack_response.rules

classification-file: /opt/etc/suricata/classification.config
reference-config-file: /opt/etc/suricata/reference.config
threshold-file: /opt/etc/suricata/threshold.config

# action-order:
#   - pass
#   - drop
#   - reject
#   - alert

# Limit for the maximum number of asn1 frames to decode (default 256)
asn1-max-frames: 256

engine-analysis:
  rules-fast-pattern: yes
  rules: yes

#recursion and match limits for PCRE where supported
pcre:
  match-limit: 3500
  match-limit-recursion: 1500

# Holds details on the app-layer. The protocols section details each protocol.
app-layer:
  protocols:
    krb5:
      enabled: no # Requires rust
    ikev2:
      enabled: yes
    tls:
      enabled: yes
      detection-ports:
      dp: "[443,444,465,853,993,995]"

    dcerpc:
      enabled: yes
    ftp:
      enabled: no
    ssh:
      enabled: yes
    smtp:
      enabled: yes
    imap:
      enabled: detection-only
    msn:
      enabled: detection-only
    smb:
      enabled: yes
      detection-ports:
          dp: 139, 445
    dns:
      global-memcap: 16777216
      state-memcap: 524288
      request-flood: 500

      tcp:
       enabled: yes
       detection-ports:
          dp: 53
      udp:
       enabled: yes
       detection-ports:
         dp: 53
    http:
       enabled: yes
       memcap: 16108864

###########################################################################
# Configure libhtp.
libhtp:
    default-config:
      personality: IDS
      request-body-limit: 2mb
      response-body-limit: 2mb
      request-body-minimal-inspect-size: 32kb
      request-body-inspect-window: 4kb
      response-body-minimal-inspect-size: 32kb
      response-body-inspect-window: 4kb
      http-body-inline: auto
      double-decode-path: no
      double-decode-query: no
    ntp:
      enabled: yes
    dhcp:
      enabled: yes
    sip:
      enabled: yes

coredump:
    max-dump: unlimited

# Suricata user pass through configuration

Thanks for sharing. From my review it appears that you did remove some valid rules which would create confusion. I haven't found a full list of what the categories are, but we can continue to tweak this.

the --simulate-ips command line option doesn't actually turn on IPS, but simulates it on for QA purposes. I don't suggest we use that as it will add confusion.

At this point having a good IDS is a good start. Will try this out.