It really shouldn't have any significant impact. On the server-side the code would check to see if that section is active (in which case it would proceed with rendering its elements) or hidden (in which case it would skip it). Best case scenario, if you just delete them completely, you save an insignificant amount of time.
Liquid rendering can significantly effect loading speeds if multiple loops are involved, or if they are done in a non-optimized manned. You can check out how much time it takes to render your liquid code (divided in nodes), by adding the Shopify Theme Inspector for your Chrome Developer tools.