Two comments

In performance tuning slower means you check I/O first. A priori it sounds like a disk efficiency problem. That slight increment indicates I/O is most likely the problem - increasingly bigger file sizes, more files, bad directory lookup performance.

Example: huge numbers of files in a single directory degrade lookup performance. This is disk hardware and filesystem dependent. We had a poster here years back who could not understand why it took ls 90 seconds to locate a file in a directory with a million emails. Workaround was/is to create a multi-branched directory tree with a lot fewer entries per directory. The find command has similar problems. The tell for this is when you get really large directory file sizes. ls -ld somedirectory PS: directory files in most filesystems are not self-reorganizing - they don't shrink. So the smoking gun does not go away usually.

Because this runs on lots of servers, you should use the very data you are getting to see if the problem is localized to a few servers, or generally spread across all of them. Not clear to me you did this.

Parallel is also obfuscating granularity for process observation. One process that does something outrageous may not show itself right away. You may have to resort to using the time command on each process and look for outliers. But first you must find at least one poster child server that clearly runs slower today compared with a while back.

Sounds like your next few Sundays (or whenever you can monopolize some servers) are spoken for.