From what I've heard, dietary cholesterol is not that much of a threat, or at least does not have as much of an impact on cholesterol values as dietary saturated fats. I assume genetics also play a big role, maybe the biggest.
Many people believe that dietary cholesterol contributes to about 1/3 of chosterol values.