Powering on main monitor moves all windows to second display

I previously had two Dell monitors connected by DisplayPort and had no issues, then I had to send one back. So I purchased a new Dell Alienware monitor and also connected it with DisplayPort and the USB-B to USB-A cable. Being a newer monitor I set it as my main.

Now whenever I power it on, it moves all windows it had to the second monitor. Also seems to be occurring if there was a display refresh or if I change settings on the monitor (like activating HDR and deactivating it). The second monitor does nothing else when I turn it on and off.

I updated the drivers both of the monitor (AW2724DM) and the GPU (NVIDIA 3070 ti).