Can I give my 13 month olive oil for extra fat?

That may sound weird, but my 13 month old refuses to eat ALL food except pureed fruits and graham crackers. I am worried he is not getting enough fat in his diet because he has a dairy allergy and can only drink soy milk, which is pretty low in fat. If I added just a tiny bit of olive oil in his baby food throughout the day to get a little bit of extra fat in him, would that be bad for any reason? He is 13 months and barely weighs 20 lbs, and is just a skinny little guy.