One major cause of this is using Docker Layer Caching when your image has many layers, you're caching many images or a combination of both. The more cached layers/images you have the longer it will take to start remote docker.

To mitigate this you can try reducing the number of layers in your image, or separate building images into multiple jobs using workflows.

Additional Resources:

Documentation: setup_remote_docker