What is the difference between the public and private custom setting?

There's no difference between public and protected custom settings in an unmanaged package scenario. Anyone with Customize Application can modify the values of the custom settings.
For a managed package, protected completely hides the values from the subscriber. This includes Apex Code, validation rules, workflow rules, flows, Visualforce pages, triggers, custom buttons and links, and any other place you might be able to access custom settings normally.
The usual case for using protected custom settings is to provide a data store that administrators cannot modify. Passwords, encryption keys, license settings, and other things that you simply do not want to allow administrators to modify should be placed in protected custom settings. This includes settings that can be "messed up" if not modified in a controlled manner.
Public custom settings allow administrators to view and modify values, as well as allow them to create code that can manipulate those values. Use public custom settings for things like "user preferences" or other non-sensitive data that isn't vital to your application's operation.