It's important to note that the resource bottleneck for a game like Cuphead is VRAM (large sprites with many unique frames, and fast gameplay requiring low latency). A different game with high resolution sprites but only 5 frames isn't going to run into that bottleneck. Nor will 1000 bullets flying around. It's not the same thing.
~
So here's an experiment I just conducted. Fresh project, latest GMS2. Room size 128x128. Texture page size 2kx2k. I add a single sprite in the corner of the room and build the executable. I take note of the VRAM usage of the executable using a process monitor, then I scale the sprite. Repeat.
Sprite size / VRAM usage
8x8 : 4096 MB
128x128 : 4160 MB
512x512 : 5128 MB
2048x2048 : 20484 MB
2048x2048 x2 frames : 36868 MB
If I take that last and second last one. 36868 - 20484 = 16384 per each 2K sprite. Now I can easily calculate how much space raw uncompressed image data (2k) takes. 4 bytes per pixel (RGBA). 4 x 2048 (rows) = 8192. 8192 x 2048 (columns) = 16777216. / 1024 (because 1 MB = 1024 KB) = 16384 on the money. Unless I'm missing something, that's pretty much proof that sprites are uncompressed in VRAM.
Does that mean games like Cuphead are impossible to make in GMS2? I honestly don't know. I think if you have a platformer where the main character is a 512x512 sprite (fairly high resolution) x 100 frames that need to be loaded at any given moment for good fluidity (remember you need transition frames), that's exactly 1GB of VRAM usage. Then there are all your enemies, items and effects. Most of them will be smaller, but there's a variety of them. Then there's the bosses. They are absolutely massive. Remember Hilda Berg's final stage? I'm pretty sure those massive sprites were no less than half the screen resolution. I'm not going to count backgrounds because in theory you could unload/load these from regular RAM in parts as you scroll. And the characters don't move that fast across the screen.
That's a huge amount of raw sprite data. If memory serves modern video card compression can compress that 8-16 times, more if you use 1bit alpha channels (which look pretty bad, but can work depending on style). Good graphics cards have 6-8 GB VRAM. Most people who game (more casually) have 1-3 GB. So, a game like Cuphead without VRAM compression, I have huge huge doubts. There's my informed opinion.
Do hope somebody proves me wrong though. I do love that game and how it looks!
Maybe by buffering only a few "future" sprites into VRAM every several frames? Would it be able to do this every half second without stutter? Move half a GB of texture data from RAM to VRAM? If the computer doesn't have more than 8GB of RAM, then in would have to move stuff off disk constantly, which would be impossible to do fast enough.
In short, texture compression seems to make games as fluid and high-resolution as Cuphead possible. I don't see a way to enable that in GMS.
I think getting half-way there (resolution and num of frames) might be doable, though.