The only thing that could be of concern is for those who have been refused entry? Who has legal authority to detain them and what if there are several onboard? Surely they can't keep track of all of them and they could easily slip off at arrival. Or do you just kick them off at the nearest station?
Obviously, a process would need to be developed. Tech has made things easier. Look at the VIA Bluetooth scanners. Some of them can now do passport scanning to catch common stuff like expiry dates, reducing issues before customs needs to clear you (with a mobile version of a more advanced scanner). VIA/Amtrak can just pre-scan you at Union while scanning your boarding pass, and inform you of issues.
Even if you boarded the train before realizing there was an issue, I would see that you could still get a choice where to disembark before the border (e.g. unstaffed West Harbour of fully staffed Niagara Falls with accesibility/amenities) with optional GO Transit "vocher" provided for free return to Toronto (Also good on any returning Amtrak or VIA trains to point of origin), good for the day. This can be a regulatory requirement and condition for 100% full, fair, humane accommodations of denied passengers, including a family saddled with disabled children and special needs.
If a specific coach has had a door unsealed to have a person (or people) disembarking witnessed by customs being present (rescanning as they disembarked to delete them from manifest), then all customs would need to do is simply get a quick passport scan again (with a little scanner type thing) to make sure only cleared individuals remained. A handheld Customs scanner would have already memorized who was on which coach, and can flag if there's a mix-up (requiring full clearance check again). Only then, the coach can be resealed. Passage between coaches would be temporarily not allowed until you passed Niagara Falls, marked via the "seals" once customs exited that particular coach.
There would need to be a process to resist tampering, mix-ups, lazy customs staff, etc. A central supervisor unit (or central server, or other) would inform if something is amiss (passport mismatch with transmitted passenger roster as an example, surplus passengers, etc) forcing a full 2-hour pause. You work the rate until it happens only once a week (or less). You pad train turn-around time to accomodate such unexpected stalls.
It's also possible a nominal number of seats could also remain empty (e.g. 8) to permit complicated situations like needing to reposition denied individuals out of cleared coaches, for operational efficiency sake (to make it more likely that customs clearing can finish before Niagara Falls, and avoid a stall). Those seats can be sold to Buffalo-NYC, just not to Toronto -- basically locked for sale on Toronto-Niagara Falls for the purpose of operational efficiencies to allow customs to keep working the train and "sealing" coaches as they left them.
_____________
Regarding complexity and technology:
Such an idea is somewhat problematic but technology has helped simplify things a lot. Bikesharing/Carsharing wasn't practical until technology made it simple. Now we've got Uber thanks to technology. We've got Google Navigation routing around traffic blockages (Cue the old 1980's AT&T advertisement mentioning a car computer with an electronic map that tells you to go around traffic -- no longer pie in the sky!). We've got Transit App, and everything. Today, we now have the tech to make on-the-fly customs clearance work in a tamper resistant way with full auditing ability, little need for customs workers to keep track thanks to transmitted manifests needing to match up with scanner data, as well as supervisors checking seals.
Tiny pocket scanners, tiny cheap doorway cameras (ability to audit of board/unboard), quick reverification processes for integrity of a coach seal / coach roster change. Yes, the snack bar might need to be closed for an hour when the doors between coaches are marked with the clearance integrity "seal". But that's better than a 2 hour wait at the border.
Now different administrations (Canada-USA) may be willing to adopt something good enough that works without inconveniencing too many people. Over the years, the border has been more strict and less strict. In the right era, a bi-national agreement then leads to on-the-fly customs clearing.
We (Canada) have the skills and technology. We are a perfect candidate country to begin trialing on-the-fly customs clearance for international rail travel. The tech already exists, some of it cheap, some of it off the shelf, some even already in use by VIA. And your existing smartphone already can
read your passport RFID chip with an app. Various militaries already has private in-house face recognition apps that can recognize you as a person of interest, directly from the camera of a hardened smartphone. What's in the airport security clearance (cameras, fingerprint scanners, passport scanners) is already 100% all mobile off the shelf now. What's stopping Customs from simply using a ruggedized/hardened Symbol handheld (Canada Customs/Homeland Security approved) ten or twenty years from now in a future era where thousands go across the border per day by rail instead of car?
If customs balks at even trying this, variants can happen, like a 2-stage clearance process. You'd do the on-the-go customs clearance as described all the above. Then you'd still do a stop at the border, but it'd be much shorter (5-minutes) by a fleet of customs officers, to check for a perfect 1-on-1 mapping with a transmitted manifest (airline style), as a supervisory scan of the earlier on-the-fly customs clearance. This redundancy may make Homeland Security accept such a plan, for example...
The off-the-shelf tech's here today and falling fast in price, and would be important at Vancouver-Seattle, Chicago-Toronto, Detroit-Windsor, Toronto-Buffalo, Montreal-NYC, etc. It is no longer an IT megaproject like it may have been considered yesterday, and a $5 onboard clearance surcharge for a few years easily pays for the capital/development costs of all this, and then continuing $5 surcharge would cover operations costs including the continual re-positioning of customs officers to points of origin.
Any incidents (fights, etc) could result in immediate stop at the nearest station (GO station, etc) and removal by local law enforcement -- just as it would for all kinds of altercations today (whether on train or at airport in the unsecure area). It happens sufficiently rarely.
Yes, customs would balk today at the concept of on-the-go clearance, but 10 or 20 years from now?