my-asu-archive/real-time-traffic-analysis-clickhouse
1
0
Fork 0
mirror of https://github.com/20kaushik02/real-time-traffic-analysis-clickhouse.git synced 2025-12-06 07:54:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
real-time-traffic-analysis-.../preprocessing
History
Kaushik Narayan R 3bf8565a16 update readme
2024-11-02 22:25:05 -07:00
..
pcap_processor.py
pcap to csv pipeline
2024-11-02 22:15:39 -07:00
README.md
update readme
2024-11-02 22:25:05 -07:00
sample_output.csv
pcap to csv pipeline
2024-11-02 22:15:39 -07:00
scratch.py
preprocessing data
2024-10-31 00:29:26 -07:00

README.md

Data filtering, preprocessing and selection for further use

  • IP packet traces are taken from here
  • Filtering
    • L4 - Limit to TCP and UDP
    • L3 - IPv6 is only around 10%, let's drop it
  • Selection of fields:
    • Timestamp
      • capture window is from 0500-0515 UTC
      • nanosecond precision, use DateTime64 data type in ClickHouse
    • IP
      • addresses - src, dst
      • L4 protocol - TCP, UDP. use Enum data type in ClickHouse
    • TCP/UDP - ports - sport, dport
    • Packet size - in bytes
  • sample_output.csv contains a partial subset of 202310081400.pcap, ~600K packets