Public key is the address in the Ethereum that is provided by you when you transfer money from your account to another. The one who is hidden behind password encryption is private key and it proves your identity to the network. The Ethereum wallet in which you are storing Ethereum is actually the private key and have access to your coins. It is password protected, because if in case someone tries to access your private key, then he/she will need your password in order to get an access of your account.

Ethereum stores its coins on the blockchain and records the amount of Ether that is stored on every address. So, if there were no private/public keys, then anybody could have used the address or have an access to your account. You do not need to secure your coins on the computer, you only have to hold the key and your coins are unlocked on the network, and you get an access to use your coins. This is how you prove your ownership to your Ethereum coins.

To have a backup of your Ethereum wallet, you need to have copy of the private key and save it as piece of paper, USB drive, or you can save it on another server. Whether you are a programmer or not, you can use the official Ethereum wallet. You just have to open the application, sign up an account, then it will prepare private and public keys for you are store them on the specific location.  

Though, you don’t make use of Ethereum wallet, even then every wallet application will store the keys. Every file in the folder will have private/public keypair. After storing this file on your computer, even if you can discard the window, your Ethers will remain safe. The private key is encrypted and it can be decrypted with the password only. Hence, it is advisable to not forget your password, as it doesn’t have ‘forgot password’ feature.