Passwordless Ssh Login to Vagrant Box from Another User

I have a new user in my vagrant box(trusty64) and I am trying to ssh into it. Instead of logging into vagrant user after vagrant up, I want to login to my username.

What I have done so far

1. Created a user in my guest machine.
2. Created ssh key in my host using ssh-keygen
3. Copied the ssh key to the guest using ssh-copy-id -p 2222 -i shash@127.0.0.1

and the part of the Vagrantfile looks like this

  config.vm.box = "ubuntu/trusty64"
  config.ssh.username = "shash"
  config.ssh.forward_agent = true
  config.ssh.private_key_path = "~/.ssh/authorized_keys"

I can use ssh -p '2222' 'shash@127.0.0.1' to login directly but when I give vagrant up I keep getting the following error

default: Warning: Connection timeout. Retrying...
default: Warning: Authentication failure. Retrying...
default: Warning: Authentication failure. Retrying...

Any help in sorting out this is really appreciated.Thanks!