Put the potatoes in a 3-quart / 2.8 L saucepan, cover with cold water by 2.5 cm / 1 inch, and add 1 tsp salt. Bring to a boil over high heat, then simmer for 12 minutes, until a knife slides into a potato with no resistance.
While the potatoes cook, whisk the yogurt, horseradish, olive oil, Dijon, 1 tbsp lemon juice, 1/2 tsp salt, and pepper in a small bowl. It should fall off the spoon in a thick ribbon. Taste and add more horseradish if you want more heat.
Slice the cucumber, radishes, and red onion. Flake the mackerel into chunks about 3 cm / 1 inch wide, checking for pin bones as you go. Cut the beets into wedges, pat them dry, and keep them separate.
Drain the potatoes and let them steam dry in the colander for 2 minutes. Toss them with about 3 tbsp of the dressing while they're still warm.
Divide the arugula among 4 bowls. Arrange the potatoes, cucumber, radishes, onion, and mackerel in sections, then place the beets last in their own spot. Spoon over the remaining dressing, scatter the capers and dill, and squeeze a lemon wedge over each bowl. Serve within 30 minutes.