Saying "I love you" is a hard thing to do. You might be worried if you are really expressing your truly feelings or if you are being pressured into it. Everyone has their own time when they think that it is appropriate to say "I love you". There is no right or wrong time. However, it is hurtful when an individual makes himself vulnerable and says, "I love you", and his words are not returned. Sometimes it may be a simple act of courtesy to return the sentiment. On the other hand you may be setting him up for false expectations about the potential for a relationship.