This is both true and false. What counts is how well the penis fits into the vagina. A long penis in a short vagina can be very uncomfortable, while a slimmer penis in a large vagina is less stimulating.
Having said that, having a small penis can often cause low self-confidence, which can make sex less enjoyable.