You always need to decompress the game either way in order to play it, so compression will mostly only matter for distribution, and as stated you can just use whatever solution you want to distribute your game in. You could even go as extreme as using something like the KGB archiver with maximum compression if you want. You just pay with time if you want to close in on the most optimal compression ratio. So if you want a smaller game overall start going through your assets and figuring out what takes those 170mb and what can be cut down on quality etc in order to bring it down. Otherwise just embrace the fact that you have a big game executable.
EDIT (for completeness).
Executable files have a
well-defined format that allows Windows to load and run their code. There are instances of executables that are actually wrapping other executables, unpacking them and delegating the execution to them at runtime (see this
Wikipedia article). However these obviously have a cost. Each time you execute them they need to decompress their entire data before being able to execute the original code. This means that in your example, if you created a packed executable using 7z format, you would need to turn those 156 MB into 170 MB every single time a user ran your game, before you even get to the stage of the game where it then actually starts loading those 170 MB of data into game format. This is/can be a heavy load time penalty for each execution. Of course you could try to create a packed executable that cached the uncompressed data, but the first execution would still suffer that initialization hit. Obviously this is not something that every GameMaker user would be happy with, so it is extremely normal that YYG doesn't do things this way.
Now, you can do this as an extra step for your game if you want, but as I mentioned above your game will always need to uncompress data before execution. It boils down to a trade-off. You can ship a highly compressed archive that gets uncompressed on install. This would save space in distribution but bloat the installation time. You could go with the packed executable that caches the uncompressed data. This delegates the penalty onto the first game execution, but it also means using a lot more data in your user's machine (since the compressed and uncompressed data would live side by side). You could create a packed executable that always unpacks the data, deleting it when the game ends in order to minimize its long-running footprint. Yet this would incur the loading time on each execution and not really save that much on space. You could instead externalize the assets from the executable and perform these same techniques on the direct game executable (or do a launcher approach where the executable is very small but it downloads all of the data from the internet when first executed). And so on.
That's why I stated that if you want to have a significant cut down in executable size start by looking at your assets. Maybe you are using high-quality audio files that could be replaced by a lower quality format that does not sacrifice that much of the playing experience. Maybe you are using oversized art assets that could be cut by 25 or 50% in size without jeopardizing quality. Maybe you are just including a lot of debugging assets that don't even get used in-game and can just be eliminated. Those things can greatly influence the final executable size. You can then further minimize for distribution if you want by applying one of the previously mentioned techniques.
