Rendered at 02:51:24 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
physix 4 hours ago [-]
Nobody uses Amex for payments, so the system isn't ever under high load.
Just kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
But still, I kind of like the design.
mixdup 4 hours ago [-]
>To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
But if the router sends to the wrong cell the cell will either send it back to be rerouted or it will fail and the router will try again (or report back the failure so upstream can try again I assume)
physix 2 hours ago [-]
That would be the good case.
But what if the cell doesn't know that, and it's holding, for example, a stale account number?
inigyou 1 hours ago [-]
Generally with a credit card, or many banking systems more generally, because they predate computers, it's possible that a charge might be accepted even if there's no knowledge whether the money is in the account. As long as the person who was supposed to have paid is identifiable, the money is taken from their account anyway in the end, and if they don't have it, they get sued and their wages garnished, and if they also don't have wages, that's a small enough percentage of people that it's part of the cost of doing business.
Insimwytim 2 hours ago [-]
Whole lot of nothing.
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
nightshift1 3 hours ago [-]
All i can see is a giant single point of failure called the Global Transaction Router.
manoDev 10 minutes ago [-]
Isn't it reassuring the entire CC business hinges on a single proprietary appliance inside a cage at some DC? :)
otterley 3 hours ago [-]
GLBs aren’t SPOFs. They are typically deployed around the world redundantly, often using Anycast IPs or using DNS geographic and failover records, and are stateless. Think AWS Global Accelerator and Route 53 as an example. The architecture diagram is a high level simplification.
wbl 2 hours ago [-]
GLBs absolutely can be SPOF for certain kind of administrative mistakes.
lokar 30 minutes ago [-]
That depends on your change control process
themafia 43 minutes ago [-]
A large portion of DNS is outside of your control. You're relying on at least two third parties you have indirect relationships with in order to work. If you're outside of the standard TLDs you've got additional social factors that can control your resolution.
Granted. It works really well in practice. It should be noted we haven't actually had the world war the Internet was designed to survive. So we're not entirely clear on the semantics of operations in unusual and unexpected configurations. I would expect DNS to be the first shoe to drop there.
mkhalil 2 hours ago [-]
microservices / clusters / zones - really all of these are other "cell-based" architectures as well. there is absolutely no written rule that a microservice was just an API or a singular service, it basically can be a independent instance that is testable/usable/gives value on itself.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
inigyou 3 hours ago [-]
403 Forbidden
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
jeremycarter 5 hours ago [-]
As Reddit already pointed out, this is nothing novel.
therealdrag0 2 hours ago [-]
Still nice to have learning resources like this pass through HN even if it’s not novel.
christophilus 4 hours ago [-]
“They reinvented Erlang OTP.” - Reddit
tomjakubowski 2 hours ago [-]
Don't know if Joe Armstrong ever said anything like it, but I would propose naming an Erlang/OTP analogue of Greenspun's tenth rule (the one about C projects containing ad-hoc, buggy implementations of Lisp) for him.
themafia 40 minutes ago [-]
To be fair Elixir shows you can just use the BEAM if you want. If you need these semantics at this level there's very few reasons not to go this route.
neerajsi 5 hours ago [-]
I wonder how they ensure durability. Is it possible that a cell going down would roll back a payment after it has occurred. Or do they depend on a non cell database?
subtlejellyfish 5 hours ago [-]
I would assume nothing related to a given transaction crosses the cell boundary.
We use a cellular architecture to help constrain the blast radius of a modular monolith. Each one of our customers lives in exactly 1 cell. Any kind of cross-customer BI/reporting happens through a data warehouse.
charcircuit 52 minutes ago [-]
This service oriented architecture except more expensive and complicated.
stevefan1999 4 hours ago [-]
Backing up would be hell
simmonmt 4 hours ago [-]
Maybe? If you assume a cell can just disappear at a moment's notice, then I'm guessing you don't even try backing it up. Whatever goes into and out of the cell (request logs and results) gets backed up, and no doubt that's more complicated than a monolithic system, but it may not be so bad assuming the replay systems and global transaction router do their thing?
kev009 5 hours ago [-]
There things are always a clusterfsck compared to the mainframe deployments.
vb-8448 4 hours ago [-]
Ahahha so true man!
Some CICS regions, a DB2 and a couple of VSAMs and that's it.
llmslave 5 hours ago [-]
American Express tech is some of the worst in the world among big companies. All of the value in the company is just in the branding. They put some work into the mobile app and the website, but other than that, its a facade.
mcintyre1994 4 hours ago [-]
A few years ago someone kept signing up for loads of bank accounts/credit cards in my name, with my address. I’m not sure what the point of it was. But while everyone else happily sent cards and stacks of welcome paperwork to me, Amex were the only one that contacted me and told me they’d detected something weird in the signup. They gave me some helpful advice to resolve that situation too.
buerkle 3 hours ago [-]
I froze my credit with the 3 big credit agencies in the US years ago when someone attempted to open multiple Dell and other company accounts in my name. Easy enough to unfreeze for a temporary period of time when I need it.
yndoendo 19 minutes ago [-]
BoA reauthorized an auto-payment card even with the card being expiration and uncorrected security code. I would call that authorized fraud by Bank of American.
This is why I find it best to declare a card stolen right before expiration or after.
jmpman 4 hours ago [-]
Having worked at Amex and other huge banks, let me assure you that there's much worse than Amex. Amex's Fraud analytics team was good. Risk was good. Ben's team is good.
tracerbulletx 3 hours ago [-]
What are you basing that statement on? It has not been by personal experience.
great_wubwub 4 hours ago [-]
Makes me a little nervous that a web page about resilience is failing to connect.
badlibrarian 4 hours ago [-]
Ah yes, the financial services company that runs a travel agency, allows me to book my hotel and rental car weeks in advance, registers a hold for incidentals for both the hotel and car when I check in, then blocks the card when I try to buy dinner that night in that same hotel due to fraud detection.
Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.
toast0 3 days ago [-]
They run their payment systems on ps3??? Somebody bought into the marketting a bit much.
rekttrader 6 hours ago [-]
So you’re telling me these cells operate independently like distributed Ethereum nodes and L2s… got it.
inigyou 3 hours ago [-]
Ethereum nodes are not independent, they are as interdependent as it's possible to be.
shwaj 11 minutes ago [-]
It's becoming a reliable heuristic that when somebody says "... got it", they probably didn't get it.
Just kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
But still, I kind of like the design.
But if the router sends to the wrong cell the cell will either send it back to be rerouted or it will fail and the router will try again (or report back the failure so upstream can try again I assume)
But what if the cell doesn't know that, and it's holding, for example, a stale account number?
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
Granted. It works really well in practice. It should be noted we haven't actually had the world war the Internet was designed to survive. So we're not entirely clear on the semantics of operations in unusual and unexpected configurations. I would expect DNS to be the first shoe to drop there.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
We use a cellular architecture to help constrain the blast radius of a modular monolith. Each one of our customers lives in exactly 1 cell. Any kind of cross-customer BI/reporting happens through a data warehouse.
Some CICS regions, a DB2 and a couple of VSAMs and that's it.
This is why I find it best to declare a card stolen right before expiration or after.
Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.