Answer
A large and growing volume of documents flagged with restricted access on the objects being viewed has been associated with this kind of slowdown in the attachment-count lookup, since that check has extra work to do per restricted document — if attachment counts are getting slower over time without an obvious configuration change, check whether the volume of restricted-access attached documents has grown significantly, and check current release notes for whether a performance improvement in this area has since shipped for your version.