Answer
Page Designer and Navigator Designer access is tied to the CustomPagesHandling and CustomNavigatorHandling projections respectively, both typically included in a broad administrative permission set — revoking those two specific projections from the permission sets assigned to the users you don't want having designer access removes their ability to open Page Designer/Navigator Designer without otherwise limiting their other administrative permissions.