Forgive my ignorance, but how exactly does Flashbake prevent bakers from ordering transactions? They still get everything sent to them, right (outside of the mempool)? Or is the data encrypted in some manner? If it is encrypted, why could a similar system not be used for transactions publicly sent to the mempool?
Flashbake does not prevent a baker from ordering transactions. It is a system to bypass the mempool. Your encryption idea sounds like timelock (which has a bug and should not be used for now).
I also wonder about the introduction of a new trusted party; what powers would theoretically be available to a hostile relay service?
A dishonest relay can see all the transactions and reorder them itself instead of simply forwarding them. But the mempool also has this problem. And this can be mitigated by running your own relay, the relay on flashbake.xyz is provided for convenience only.
Lastly, could a hostile party not spam the relay service with a high number of low-fee transactions, since that party doesn’t have to pay fees for transactions not included in the block?
Indeed, or you can also spam the baker’s Flashbake endpoint directly. But spam can be mitigated by prechecking every operation before accepting or fowarding it, much like Octez does it.
