I agree. Stated another way, imagine the trolley is headed towards 5 people, and you have the power to pull the lever to divert it to a path where there are no people. Even if someone tied those 5 to the tracks with the intent to kill, your failure to save their lives (at no additional cost to others) is widely regarded by most systems of ethics/morality as a moral failing. Yes, the person who tied the tracks bears blame, but so does the person who could’ve easily saved them but chose to let them die.
Any idiot can build a bridge that stands. It takes an engineer to design a bridge that barely stands.