With most things, trust is established "by proxy" (by trusting someone who trusts whatever you worry about).
In free software world, we generally trust that free software is well-meaning and contains no backdoors because we assume that someone else has reviewed the code: we don't go reviewing each and every library or app we use.
Sometimes that trust is misplaced, particularly with small, unused apps or libraries, but in a grand scheme of things, it works.
Elections are slightly different in that you want precision and guarantees. I think electronic voting can and should be introduced with a fully open source stack even for important votes, but it should never be mandatory: in theory, for those who understand the process, verifying their vote is much easier electronically. This would enable trust-by-proxy to work as well for the technically inclined.
There is the trouble of verifying that the software any voting system is running matches the source code you are given access to. In that sense, having public access to a full database of votes (or at least electronic ones) would help quell that concern as well — at the very least, each competing political party could run their own system to verify your vote.
Still, I think the easiest way to manipulate votes today, and how it's usually done in democracies, is by media manipulation (by selective reporting or over-reporting to drive a narrative), and no voting system can help with that :)
In the grand scheme of things, almost no computer system can withstand a focused, persistent attack by a nation-state level attacker. Air-gaps aren't enough [1], being extremely widely-used open-source isn't enough [2], even being formally verified isn't enough, because even a formally verified compiler can have backdoors [3]. And we haven't even touched whether you can trust the silicon it's running on [4].
And even in the absurdly unlikely case where a secure voting system is implemented, voters will connect to it with laughably insecure personal computers, that can alter their votes at will. I don't want to have to trust Microsoft and Intel's [5] benevolence to not abuse their root-or-lower level access to devices to alter votes as they are cast.
> but it should never be mandatory
So the 1% of voters that understand how insecure online voting is vote in person - or, let's be extremely generous and say 60% vote in person, while 40% have their votes altered by a hostile nation state (this includes the nation state that made their voter's operating system of designed or fabbed their CPUs). There's almost no election you can't swing if you control 40% of all votes.
With the prize being clandestine control of a country, someone is bound to make the effort. And then what will you do? Vote to return to paper ballots? They are imperfect, but attacks on them don't stealthily scale to a whole country.
I won't discount the risks associated with electronic voting and electronic machines in general, but 40% of "wrong" votes are almost impossible to miss with the most rudimentary checks. Eg. if 60% of paper ballots have a ratio swinging 4 to 1 one way, and 40% of electronic votes have a ratio swinging 4 to 1 the other way (this is commonly done in any statistical endeavour to detect correlations, and much smaller effects can be noticed), any observer and participant in the elections can notice that when looking at the final tally, as long as there is a breakdown by the type of vote submission as well (which is already done for things like mail-in votes).
And if a vote is swinged completely against the actual public vote, you'll only end up with either a quick drop of the electronic system (which all the IT providing companies like MS or Intel would hate to see happen, which is an incentive for them to not interfere), or angry mobs hitting the streets.
Basically, what you are left with is influencing the outcome so that what would have been a 51-49 winning party is now a 49-51 losing party. Yet there are other "unfair" ways to achieve that even today that are not as involved as hacking the entire electronic voting system (as I mentioned, media manipulation being the key one, or providing benefits to vote a particular way or...).
So yes, ultimately, everything can go wrong, but no, it won't because there are safeguards against that happening regardless of the method of voting.
You underestimate how large a manipulation could go undetected. [1] shows an election with a 28-41 win for Patrick, while the poll showed a 37-31 win for Dewhurst. That's a 10% absolute swing, and 25% relative. And the less attention an election gets from pollsters, the bigger the margin for error and manipulation.
Changing too many votes, to make people doubt the legitimacy of the election, is itself a powerful attack to sow division in a country. I'm sure I don't need to cite examples...
And then what is even the point of formal voting, if, when done online, it's a centralized way for an adversary to change results, and is only legitimized by reproducing results of informal polls (conducted by public research firms we trust, of course...) We may as well skip elections and use the polls directly.
I agree there are many other ways to manipulate voters, so let's not expand that list further.
In my example, a formal vote is partially done electronically, and partially through paper votes: large discrepancies in results between those two groups could be indicative of interference (small variation is to be expected). I am not talking of unofficial polls which can never produce a representative sample of people.
Point of formal voting is to elect a government that the majority of voters prefer. Many countries have a problem with voter turnout, so you rarely get any one party or coalition winning more than 20-30% of the eligible votes, yet they get majority control of the government.
Point of electronic voting should be to increase the turnout without decreasing the trustiness. It's not an easy problem to solve, but it's not insurmountable.
Electronic votes can be decentralized as well: final tallying is centralized with any approach, but it can similarly remain open to questioning.
In free software world, we generally trust that free software is well-meaning and contains no backdoors because we assume that someone else has reviewed the code: we don't go reviewing each and every library or app we use.
Sometimes that trust is misplaced, particularly with small, unused apps or libraries, but in a grand scheme of things, it works.
Elections are slightly different in that you want precision and guarantees. I think electronic voting can and should be introduced with a fully open source stack even for important votes, but it should never be mandatory: in theory, for those who understand the process, verifying their vote is much easier electronically. This would enable trust-by-proxy to work as well for the technically inclined.
There is the trouble of verifying that the software any voting system is running matches the source code you are given access to. In that sense, having public access to a full database of votes (or at least electronic ones) would help quell that concern as well — at the very least, each competing political party could run their own system to verify your vote.
Still, I think the easiest way to manipulate votes today, and how it's usually done in democracies, is by media manipulation (by selective reporting or over-reporting to drive a narrative), and no voting system can help with that :)