While 20% down is often ideal for conventional loans to avoid PMI, it's not always necessary. FHA loans require as little as 3.5% down, and VA and USDA loans can be 0% down. The median down payment for first-time buyers is often much lower than 20%.