IPFIX (Netflow v10) logging mechanism

[Mike K.]

Hi all,
I work for a network management and reporting company that
specializes in NetFlow collection and analysis. We have found that a
majority of commercial vendors of firewalls and upper OSI network gear
are now in the process of developing IPFIX logging mechanisms. I am a
big fan of SQUID and I would love to see it implemented here as well.
It seems to me that it would be good for the project if anyone was
interested in implementing IPFIX logging in squid. This is a
completely open standard that was derived from Cisco's Netflow v9. It
might even be fairly straightforward to add this on using LIBIPFIX.
http://sourceforge.net/projects/libipfix .

My company is not in a position to develop IPFIX support in squid, but
we can be a resource to make sure that it is being implemented
properly. I guess I wanted to throw this out there to see if anyone is
interested in this feature.

Thanks,
Mikek

[Amos Jeffries]

Post by Mike K.
Hi all,
I work for a network management and reporting company that
specializes in NetFlow collection and analysis. We have found that a
majority of commercial vendors of firewalls and upper OSI network gear
are now in the process of developing IPFIX logging mechanisms. I am a
big fan of SQUID and I would love to see it implemented here as well.
It seems to me that it would be good for the project if anyone was
interested in implementing IPFIX logging in squid. This is a
completely open standard that was derived from Cisco's Netflow v9. It
might even be fairly straightforward to add this on using LIBIPFIX.
http://sourceforge.net/projects/libipfix .
My company is not in a position to develop IPFIX support in squid, but
we can be a resource to make sure that it is being implemented
properly. I guess I wanted to throw this out there to see if anyone is
interested in this feature.
Thanks,
Mikek

Hi Mikek,

After a very brief look at what IPFIX is I fail to see how it would be
of much use having in Squid. It appears to me to be a Layer-4
measurement protocol, with Squid being layer-7.

At this point the only information I can imagine Squid adding to the
IPFIX data is what inbound client TCP/UDP flow triggered a particular
outgoing TCP/UDP flow. That is only possible after the Squid comm layer
is restructured.
The future SMP support may bring other details about internal machine
process flows, but that seems not to be entirely IPFIX relevant.

Did you have any existing ideas on what information Squid could provide
over IPFIX?

Amos

[Amos Jeffries]

Hello Amos,
The Purpose of implementing IPFIX in Squid would be a logging
mechanism similar to using syslog to collect log data on remote
machines.
I think that the way you are seeing IPFIX is presented in a way that
looks like it is limited to Layer4 data. The reason for this is that
IPFIX has evolved from Cisco's NetFlow which was limited to layer 4
data.
IPFIX is a protocol that allows any type of data to be packaged and
sent to a collector. A template is sent along side the packaged data
that tells the collector what to do with it.
For instance, a very basic data export template out of squid would
+-----------------+-----------+----------------+------------------------------+
|source addr | bytes | code + HTTP URL |
+-----------------+-----------+-----------------+-----------------------------+
Each IPFIX packet can contain about 25 "flow" records as opposed to
Syslog's limit of 1 per packet.
The template system has big advantage over syslog because any IPFIX
compliant collector should be able to collect, properly parse and
store this data immediately for reporting purposes. This completely
eliminates the need to parse a logfile to generate reports.
There are several vendors that are implementing IPFIX to export.
-One such product is nProbe http://www.ntop.org/nProbe.html . As you
can see, this product generates records and export things like VOIP,
SMTP, latency, jitter and (soon) HTTP information.
-A major firewall vendor we are working with will also be implementing
the export of this higher level data via IPFIX.
Anything that can be logged by squid can be exported as IPFIX data and
in turn be available for reporting in near realtime. This seems to be
a real advantage over how reporting needs to occur now.
I hope I have explained this properly. Please let me know if you have
any questions about IPFIX.
Warm regards,
Mike Krygeris

Ah okay, that makes more sense than the search results.

Yes it looks like a potentially useful extra loging module if someone
wants to code it.

Amos