This error typically occurs when the Presto project is too large, or the resolution is set too high.

Breaking up the project to smaller parts or lowering the resolution would help. To change the resolution, go to the Edit tab then select Project Options tab, and change the Output Size.