mod_vpp: don't re-sample PSAMPLE when osIndex=on

[ritvikiscool9]

When osIndex=on, VPP writes the Linux ifIndex into the PSAMPLE metadata and mod_psample already emits the flow sample in that namespace. mod_vpp's evt_psample() then calls takeSample() a second time, so every flow sample is duplicated at the collector (2x) while the VPP plugin's own counter shows 1x. Return early in that mode and let mod_psample own the flow sample; mod_vpp still provides VPP interface counter samples via its netlink channel.

[sflow]

What does /etc/hsflowd.conf look like in this setup? It seems like there might be a misunderstanding about the use of the PSAMPLE group numbers.

For a regular SONIC switch the samples appear on PSAMPLE group=1 (for ingress) or 2 (for egress). In that case mod_psample will announce the sample via HSPEVENT_PSAMPLE and then go ahead and call takeSample() itself.

However when VPP sends samples they go to PSAMPLE group=3 (for ingress) and 4 (for egress). In that case mod_psample will still announce the sample via HSPEVENT_PSAMPLE, but it will not call takeSample(). Instead mod_vpp will pick them up and then call takeSample() itself after adjusting the port numbers.

(Later in the pipeline, mod_sonic will translate those numbers from the Linux namespace to the SONiC namespace, but we don't need to worry about that here.)

When hsflowd is compiled with FEATURES=SONIC it automatically adds the equivalent of this to the config:

   psample {group=1 ingress=on egress=on}

Plus the sonic{...} and dropmon{...} config too.

None of that needs to change. I think for sonic-vpp you just need one more line in /etc/hsflowd.conf:

   vpp { osIndex=on }

Is this config step something I can find in the sonic-buildimage repo? I'm not sure where to look.

[sflow]

I don't think you need to change anything in the hsflowd source code. I think you just need to get the right settings into your /etc/hsflowd.conf config file. Where is it defined? I don't know what repo to look in. Please send a link to the place where hsflowd is compiled with the VPP feature enabled, and the place where the /etc/hsflowd.conf file is set up.

[ritvikiscool9]

@sflow

Everything is in sonic-buildimage, under src/sflow/hsflowd/:

• VPP features enabled: patch/0002-enable-mod-vpp.patch sets the build to FEATURES="SONIC VPP"
• hsflowd.conf: is hsflowd.conf.sonic, patch/0003-enable-vpp-stanza.patch just adds one line, vpp { osIndex=on}. psample is left at the default.

Here is the live /etc/hsflowd./conf:

cisco@sonic:~$ docker exec sflow cat /etc/hsflowd.conf
│# hsflowd configuration file
│# http://sflow.net/host-sflow-linux-config.php
│
│sflow {
│  # mod_sonic, mod_psample and mod_dropmon are loaded automatically
│  # when hsflowd is built with FEATURES=SONIC, so only uncomment here
│  # if you need to override defaults. This file may be overwritten
│  # again, so when testing you might want to copy it first.
│  # For example:
│  # cp /etc/hsflowd.conf /etc/hsflowd_test.conf
│  # <edit /etc/hsflowd.conf to adjust settings>
│  # gdb --args hsflowd -f /etc/hsflowd_test.conf -ddd
│  
│  # sonic { waitReady=300 setifalias=on setifname=on suppressOther=on }
│  # psample { group=1 ingress=on egress=on }
│  # dropmon { start=off sw=off hw=off hw_unknown=off
│  #           force=on
│  #           sw_passive=on hw_passive=on
│  #           limit=100 max=100000 max_trip=2 }
│
│  # Load mod_vpp so VPP hw_if_index values in packet/counter samples are
│  # translated into the Linux (netdev) ifIndex namespace. osIndex=on is the default.
│  vpp { osIndex=on }
│  
│  # ====== detect new interfaces ======
│  refreshAdaptors=60
│  # ======  Agent IP selection ======
│  # determined by sonic config DB
│  # ====== Sampling/Polling/Collectors ======
│  # determined by sonic config DB
│  # ====== collectors ======
│  # determined by sonic config DB
│  # ====== Docker container monitoring ======
│  # docker { }
│}

For reference on the behaviour: with this config the collector shows ingress ~2x (VPP's own packet samples sent counter is 1x). Setting psample { group=1} gives 0 packets.

[sflow]

I can't seem to find the string "osindex" in any of the sonic-net repos, so please let me know where I should look?

What you are saying makes it sound as though hsflowd.conf has been set up with "psample{group=3}" when it should have "psample{group=1}".... or no entry for psample{} at all because "psample{group=1}" is implicit when you compile for SONIC. So if I guessed right then all you need to do is remove the "psample{group=3}" line.

[sflow]

OK, that hsflowd.conf looks right, so there must be something else going on. If you can set a gdb breakpoint here:

host-sflow/src/Linux/mod_psample.c

Line 372 in 41f10e6

if(psmp.grp_no

can you confirm that the samples coming from vpp are on psample group 3 or 4, and are therefore given no further consideration by mod_psample?

[sflow]

As a possibly easier shortcut, you could also do this (it runs a second hsflowd at debug level 3) so you can just grep for the debug line that prints the psample group at mod_psample.c:362):

docker exec sflow -it /bin/bash
hsflowd -dddd | grep psample

Running a second hsflowd is a little reckless because it will also send duplicate samples. If that's not OK you can also copy /etc/hsflowd.conf to /tmp/hsflowd.conf, edit it down to just one "collector { ip=127.0.0.1 }" and then run it with hsflowd -dddd -f /tmp/hsflowd.conf)

[ritvikiscool9]

-dddd shows the full picture, samples arrive on three groups (1,3,4), here is a sample of the output:

mod_psample:grp=1
mod_psample:grp=1 in=12 out=0 proto=2048 n=5000 seq=1373 drops=
mod_psample:psample sampling N (5000) != configured N (0)
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=3
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=3
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=4
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=4
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=1
mod_psample:grp=1 in=12 out=0 proto=2048 n=5000 seq=1374 drops=
mod_psample:psample sampling N (5000) != configured N (0)
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=3
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0
mod_psample:grp=3
mod_psample:processNetlink_PSAMPLE (type=36) CMD = 0

So ingress ie being sampled on both group 1 and group 3: mod_psample exports the group-1 copy and mod_vpp exports the group-3 copy. Thats the 2x on ingress. Egress only appears on group 4 (mod_vpp), so it stays 1x.

Any idea where the group-1 ingress samples are coming from in addiiton to VPP's group-3? (in=12 looks like the Linux ifindex so maybe a kernel/tap-side sampler alongside VPP's own?)

[sflow]

That's interesting. I wonder where it is coming from.

For hsflowd itself to have caused it, there would need to be a dent{} module. The mod_dent.c code can configure kernel sampling using calls to tc(1). But that is not even compiled for this profile.

I wonder, the sflowmgr script inside the sflow container has a script called "psample" that is supposed to configure the hardware sampling via the SAI API. Maybe it has a mode where it detects there is no ASIC present, and falls back on configuring kernel sampling with tc(1)? I have never looked at that script, so I don't know what is in there.

Naturally we'll want to find and eliminate that psample configuration for the sonic-vpp platform, but as a quick test you could try adding "psample{group=5}" to your hsflowd.conf file. I think that should override the implicit "psample{group=1}", and should therefore cause mod_psample to ignore the rogue group=1 samples.

[ritvikiscool9]

`psample {group 5 ingress=on egress=on} fixes the 2x on sonic-vpp.

I re-ran with aligned window (a 30s foreground sflowtool capture backtracked by VPP's packet samples sent counter):

• VPP delta = 74 (samples the dataplane generated)
• collector = 55 (ingress 27 + egress 28)

So the collector now tracks the VPP counter at 1x instead of 2x

That confirms the root cause you pointed to: ingress was being emitted on both PSAMPLE group 1 and group 3. mod_psample exported the group-1 copy and mod_vpp exported the group-3 copy, giving the two samples per ingress packet. Moving mod_psample to group 5 makes it ingore both the stray group-1 and VPP's 3/4, so mod_vpp is the sole owner of 3/4.

My full test:

cisco@sonic:~$ docker exec sflow grep -n 'psample {' /etc/hsflowd.conf 
15:  psample { group=5 ingress=on egress=on }
cisco@sonic:~$ show sflow | grep 6344 || (sudo config sflow collector add insp 127.0.0.1 --port 6344; sudo config save -y)
    Name: insp                IP addr: 127.0.0.1       UDP port: 6344   VRF: default
cisco@sonic:~$ docker exec sflow pkill -f sflowtool; sleep 2
cisco@sonic:~$ B=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ docker exec sflow timeout 30 sflowtool -p 6344 -l > /tmp/s.txt   
cisco@sonic:~$ A=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ echo "VPP delta = $((A-B))"
VPP delta = 74
cisco@sonic:~$ echo "collector total   = $(grep -c '^FLOW' /tmp/s.txt)"
collector total   = 55
cisco@sonic:~$ echo "Eth8 ingress: $(grep '^FLOW' /tmp/s.txt | awk -F, '$3==9&&$4==0' | wc -l)"
Eth8 ingress: 27
cisco@sonic:~$ echo "Eth8 egress : $(grep '^FLOW' /tmp/s.txt | awk -F, '$4==9' | wc -l)"
Eth8 egress : 28
cisco@sonic:~$ show sflow interface

sFlow interface configurations
+-------------+---------------+-----------------+----------------------+
| Interface   | Admin State   |   Sampling Rate | Sampling Direction   |
+=============+===============+=================+======================+
| Ethernet0   | up            |           40000 | tx                   |
+-------------+---------------+-----------------+----------------------+
| Ethernet4   | up            |            1000 | rx                   |
+-------------+---------------+-----------------+----------------------+
| Ethernet8   | up            |            5000 | both                 |
+-------------+---------------+-----------------+----------------------+

On a separate terminal I sent traffic:

cisco@sonic:~$ sudo ping -f 10.0.2.2 >/dev/null 2>&1 &                
[1] 279570                                                            
cisco@sonic:~$ sudo pkill -f 'ping -f'                                
[1]+  Terminated              sudo ping -f 10.0.2.2 > /dev/null 2>&1  

[ritvikiscool9]

Small update, I ran the test with just group 1:

• group=5: collector 55 vs VPP 74 -> 1x
• group=1: collector 55 vs VPP 75 -> 1x

So in the current state both read 1x - the 2x is intermittent, not a fixed property of the config group. That lines up with the rogue group-1 sample that's only active some of the time, when it's off, ingress rides only VPP's group-3 and mod_vpp exports it once regardless of the psample group. The earlier -dddd capture did show ingress on both group 1 and group 3, so now I'm looking for what intermittently enables that group 1 sample on sonic-vpp, following your lead on the psample helper's ASIC-absent fallback.

[ritvikiscool9]

Correction to my earlier comment, the "both read 1x, so it's intermittent" conclusion was wrong. That test was on a both-direction port. I re-ran a controlled A/B on an ingress-only port (Eth4, rx), comparing the collector against VPP's packet samples sent counter (the reliable 1x reference), changing only eh psample group:

• group 1 (default), VPP delta: 213, collector: 332, ratio: 1.56x
• group 5, VPP delta: 171, collector: 147, ratio 0.86x (~1x)

Same port, same metric, only the group changed, so the duplication is real and reproducible and group=5 fixes it. Egress on Eth4 was 0 throughout (rx-only), and all ingress was on in=5, so this is a clean measurement.

This matches the -dddd capture: ingress is emitted on both group 1 and group 3 - mod_psample exports the group-1 copy, mod_vpp the group-3 copy (2x ingress). Egress only on group-4 (1x). Vacating group 1 leaves mod_vpp sole owner of 3/4 -> 1x.

So group=5 is a good workaround, but it isn't the real fix - group 1 is the SONiC sampling chanel (real ASICs deliver on group 1/2), so we can't just remove it. The actual issue is two ingress samples running at once on sonic-vpp: VPP-native (group 3, via mod_vpp with ifinindex translation) and a host/kernel side sampler emitting group 1 with the raw Linux ifindex (in=12). Egress only has the VPP one (group 4), which is why only ingress doubles. The permanent fix is to stop the redundant group-1 sampler on the VPP platform and keep VPP-native 3/4 + mod_vpp translations, rather than touching group 1 globally.

[sflow]

Thank you for sharing the details as you track this down. Another suggestion would to to query for the tc(1) config to see if that was the mechanism that was used to add the rogue sampling on group 1.

(And please respond to my comments on the patch that adds per-interface sampling to the vpp dataplane. I think that one is more important than this one, and will be more work to fix.)

[ritvikiscool9]

I will try the query for tc(1).

Regarding the comment you left, could you please share the link to the PR you left a comment to, I'm not seeing it in any PRs, so I must be missing one.

[ritvikiscool9]

The tc(1) query did reveal a lot. tc filter show on the taps inside syncd show:

│cisco@sonic:~$ docker exec syncd bash -c 'for d in $(ls /sys/class/net); do out=$(tc filter show dev $d parent ffff: 2>/dev/null); [ -n
│ "$out" ] && echo "=== $d ===" && echo "$out"; done'
│=== Ethernet4 ===
│filter protocol all pref 49152 matchall chain 0 
│filter protocol all pref 49152 matchall chain 0 handle 0x1 
│  not_in_hw
│        action order 1: sample rate 1/1000 group 1 pipe
│         index 4 ref 1 bind 1
│=== Ethernet8 ===
│filter protocol all pref 49152 matchall chain 0 
│filter protocol all pref 49152 matchall chain 0 handle 0x1 
│  not_in_hw
│        action order 1: sample rate 1/5000 group 1 pipe
│         index 2 ref 1 bind 1

So:

Ethernet4:  action sample rate 1/1000 group 1
Ethernet8:  action sample rate 1/5000 group 1

ifindex 12 = Ethernet8, matching the earlier in=12 group 1 rate=5000 psample line. These are added by VirtualSwitchSaiInterface::preSetPort() in sonic-sairedis, which on INGRESS_SAMPLEPACKET_ENABLE run tc .. matchall action sample rate <N> group 1 on the tap - with group hardcoded to 1, and ingress-only (no egress branch). This is exactly why ingress doubles and egress stays 1x. SO on sonic-vpp this tc sampler runs in parallel with VPP-native group-3 sampling. Fix is to skip that tc block on the VPP platform, group 1 stays untouched for real ASIC SONiC.

[ritvikiscool9]

@sflow

I've done some more testing with tc(1) case:

The group-1 sampler comes from sonic-sairedis, not hsflowd. VirtualSwitchSaiInterface::preSetPort(), on INGRESS_SAMPLEPACKET_ENABLE runs on the tap:

tc filter add dev <tap> parent ffff: matchall action sample rate <N> group 1

group is hardcoded to "1" and it's ingress-only, exactly why ingress doubles and egress stays 1x. I ran a test where I deleted the tc sampler on Eth8, nothing else changed:

• Before (tc on): ingress: 54, egress: 27
• After (tc off): ingress: 31, egress 28

Ingress -> 1x, egress unchanged. This removes only the duplicate, no egress changed. The real fix belongs here, gate the tc sampler off on the VPP platform, (VPP samples natively on 3/4), leaving group 1/2 intact for real ASIC SONiC.

The full test output:

cisco@sonic:~$ docker exec syncd tc filter show dev Ethernet8 parent ffff:
filter protocol all pref 49152 matchall chain 0 
filter protocol all pref 49152 matchall chain 0 handle 0x1 
  not_in_hw
        action order 1: sample rate 1/5000 group 1 pipe
         index 1 ref 1 bind 1

cisco@sonic:~$ docker exec sflow pkill -f sflowtool 2>/dev/null; sleep 2
cisco@sonic:~$ docker exec sflow timeout 8 sflowtool -p 6343 -l > /tmp/pre.txt; echo "pre FLOW=$(grep -c '^FLOW' /tmp/pre.txt)"
pre FLOW=19
cisco@sonic:~$ docker exec sflow pkill -f sflowtool 2>/dev/null; sleep 2
cisco@sonic:~$ B=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ docker exec sflow timeout 30 sflowtool -p 6343 -l > /tmp/before.txt
cisco@sonic:~$ A=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ echo "BEFORE (tc group-1 sampler ON):"
BEFORE (tc group-1 sampler ON):
cisco@sonic:~$ echo "  VPP delta    = $((A-B))"
  VPP delta    = 75
cisco@sonic:~$ echo "  Eth8 ingress = $(grep '^FLOW' /tmp/before.txt | awk -F, '$3==9&&$4==0'|wc -l)"
  Eth8 ingress = 54
cisco@sonic:~$ echo "  Eth8 egress  = $(grep '^FLOW' /tmp/before.txt | awk -F, '$4==9'|wc -l)"
  Eth8 egress  = 27
cisco@sonic:~$ docker exec syncd tc qdisc delete dev Ethernet8 handle ffff: ingress
cisco@sonic:~$ docker exec syncd tc filter show dev Ethernet8 parent ffff:  
cisco@sonic:~$ docker exec sflow pkill -f sflowtool 2>/dev/null; sleep 2
cisco@sonic:~$ B=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ docker exec sflow timeout 30 sflowtool -p 6343 -l > /tmp/after.txt
cisco@sonic:~$ A=$(docker exec syncd vppctl show sflow | grep 'packet samples sent' | grep -oE '[0-9]+')
cisco@sonic:~$ echo "AFTER (tc group-1 sampler REMOVED):"
AFTER (tc group-1 sampler REMOVED):
cisco@sonic:~$ echo "  VPP delta    = $((A-B))"
  VPP delta    = 73
cisco@sonic:~$ echo "  Eth8 ingress = $(grep '^FLOW' /tmp/after.txt | awk -F, '$3==9&&$4==0'|wc -l)"
  Eth8 ingress = 31
cisco@sonic:~$ echo "  Eth8 egress  = $(grep '^FLOW' /tmp/after.txt | awk -F, '$4==9'|wc -l)"
  Eth8 egress  = 28

[sflow]

When the sonic-sairedis script injects the tc(1) config for psample group 1, does it say why? Is it just so that a software-switch deployment such as sonic-vm can generate sFlow, or is there a corner-case that it is addressing? Put another way, is there likely to be any port on a sonic-vpp switch that should support packet-sampling but is not actually a port in the VPP dataplane? (I'm thinking probably not, but wanted to bring it up just in case).

I moved the per-interface sampling comment to the PR. Let me know if you don't see it.