Yes, I use. The first weighted clothes we can buy has a base weight value of 5(If I'm not mistaken), so every fight you win you'll gain these extra +5 AP.
Also, there are others sources to increase this: Each point in athletics increase the gained AP by +2. Muscular Body type increases +1. Earth Element +1. Midorikatai Style +1 for every level of mastery...etc.
The Waverunner Armour has a base value of 9, just like the Lantern King Weighted Clothes.
There is no real harm in using the weighted clothes.
The point is that I don't focus in Athletics in the early game. I keep it a innate 1 until the Daichi fight where you can up it to a 2 Innate. And in Nanhou indeed I start to focus in Athletics(+2 Through Weapons factotum seals, +1 with auspicious seals. And if you want to spend some path manual in it, in Nanhou you can receive 3).
The points someone usually invest in athletics I use to up persuasion, after all, this is a cycling build, so you need money.
No, this 1 Agility I'm considering with the Weighted clothes already, so 2 Agility in total.
And well, I'm a Kuro player too, the point of Kuro is that the style already is focused in Agility by itself(Even so it helps a lot in Agility tests, especially in Nanhou).