mirror of
https://github.com/20kaushik02/real-time-traffic-analysis-clickhouse.git
synced 2025-12-06 11:14:06 +00:00
17 lines
640 B
Markdown
17 lines
640 B
Markdown
# Data filtering, preprocessing and selection for further use
|
|
|
|
- IP packet traces are taken [from here](https://mawi.wide.ad.jp/mawi/samplepoint-F/2023/)
|
|
- Filtering
|
|
- L4 - Limit to TCP and UDP
|
|
- L3 - IPv6 is only around 10%, let's drop it
|
|
- Selection of fields:
|
|
- Timestamp
|
|
- capture window is from 0500-0515 UTC
|
|
- nanosecond precision, use `DateTime64` data type in ClickHouse
|
|
- IP
|
|
- addresses - src, dst
|
|
- L4 protocol - TCP, UDP. use `Enum` data type in ClickHouse
|
|
- TCP/UDP - ports - sport, dport
|
|
- Packet size - in bytes
|
|
- `sample_output.csv` contains a partial subset of `202310081400.pcap`, ~600K packets
|