Answer
This has been observed specifically with larger permission sets. Two workarounds: re-import the same XML file again using the merge option, without cleaning up the failed partial import first — leaving the partially-imported state in place lets the merge import pick up faster since it effectively only needs to import whatever didn't succeed the first time. Alternatively, export the permission set as a SQL file instead of XML and import that via the command line, which has proven more reliable for larger permission sets than the XML-based import through the client.