No you can’t restrict permission, permission sets extend users’ functional access without changing their profiles.
A permission set is a collection of settings and permissions that give users access to various tools and functions. The settings and permissions in permission sets are also found in profiles
No, you can not restrict permission for users using the permission set. Permission Sets in Salesforce provide additional access to the users without changing their profile. It is a collection of settings and permissions.