|
Post by MoneyThing on Oct 13, 2016 12:43:51 GMT
No doubt I will need to get The Shuang to translate it for me anyway...although perhaps it will go through their PR department to strip out any complex language for wider consumption amongst non-techies. Happy to share the findings upon request once I have had a chance to read it (and subject to any confidentiality clauses). Regards, Ed If it goes thru PR it might tell you something....if it goes via their lawyers then no chance... I use to look after the operational side of a payment gateway business processing millions of online card transactions on behalf of UK businesses. Part of my remit was to communicate with all our customers (the merchants as well as the press), when there were any disruptions to the service. Whilst some of the time it was our fault, other times it was some other part of the 'network' (e.g. problems with one of the card issuing banks or the merchant acquiring banks). If the latter, we would need sign off from their in-house lawyers before we could publish anything and by the time they had gone over my draft nearly everything of substance had been stripped out.
|
|
treeman
Member of DD Central
Posts: 1,026
Likes: 557
|
Post by treeman on Oct 13, 2016 15:12:50 GMT
Dunno if it's related or just the site being mobbed but it was REALLY SLOW getting pages loaded and money in at 4pm today ................ No big deal as there was plenty on offer / low bid limits - would have stuffed any attempt to be FFF though !!! <EDIT> MoneyThing ShuangThing - 16:13 and back to normal loading etc - guess the demand was pretty high at 4.
|
|
dermot
Member of DD Central
Posts: 862
Likes: 517
|
Post by dermot on Oct 13, 2016 15:15:19 GMT
I had a great of difficulty getting a couple of loans in just now - I tried from both a 4G cellular connection and cable - both slow to load and dropping connections and saying site unavailable.
Did manage to get what I wanted, but a little frustrating.
Looks like the same slowdown gremlins that hit another P2P site a few days ago moved onto other pastures...
|
|
dermot
Member of DD Central
Posts: 862
Likes: 517
|
Post by dermot on Oct 13, 2016 15:16:14 GMT
Huh, yes - snappy as ever now ....
|
|
|
Site Down
Oct 13, 2016 15:17:48 GMT
via mobile
Post by geraldine1210 on Oct 13, 2016 15:17:48 GMT
It kept collapsing on me.
I think hour loans at the same time was not a good idea. As mentioned above, luckily there was plenty for all.
|
|
|
Post by GSV3MIaC on Oct 13, 2016 15:20:56 GMT
It's purely down to FFF overload at 4pm I think. This sort of FFF activity DOES NOT SCALE (ask FC, heck ask anyone who deals in real time systems, queuing theory etc.) .. and it's going to get worse unless something radically changes. Releasing several loans at once didn't help, but the real problem is ever increasing numbers of lenders (&/or bots) hitting the same page (which the server has to keep updated, lest it sells what it ain't got) repeatedly at 4pm (rinse and repeat until they either get lucky or give up).
|
|
|
Site Down
Oct 13, 2016 15:30:20 GMT
via mobile
Post by geraldine1210 on Oct 13, 2016 15:30:20 GMT
It's purely down to FFF overload at 4pm I think. This sort of FFF activity DOES NOT SCALE (ask FC, heck ask anyone who deals in real time systems, queuing theory etc.) .. and it's going to get worse unless something radically changes. Releasing several loans at once didn't help, but the real problem is ever increasing numbers of lenders (&/or bots) hitting the same page (which the server has to keep updated, lest it sells what it ain't got) repeatedly at 4pm (rinse and repeat until they either get lucky or give up). There should not be any bots. I think the loans should have been staggered. Either do them on two days, or with maybe a two hour gap in starting time from first two to second two.
|
|
investibod
Member of DD Central
Posts: 288
Likes: 152
|
Post by investibod on Oct 13, 2016 15:42:22 GMT
I think the loans should have been staggered. Either do them on two days, or with maybe a two hour gap in starting time from first two to second two. I do not think the gap needs to be so long. The slowdown is only while there is a burst of FFF. 15 minutes between launches should be plenty. I remember a similar thing happening a while ago and Ed said that Shuang had rewritten things to stop it happening again. I know that for the loan launched immediately after that there was no slowdown. Maybe the architecture needs another tweak/redesign now that >2000 users could all be hitting the refresh at the same time.
|
|
|
Post by barberbookie on Oct 13, 2016 15:50:32 GMT
That is the worse i have seen it. Took about 10 mins to get involved in the 4 loans. Constantly crashing .
|
|
ali
Member of DD Central
Posts: 313
Likes: 311
|
Post by ali on Oct 13, 2016 15:53:31 GMT
To be fair, I had absolutely no problems. At around 4pm it was slow (taking about 1m to load the loans page), but it never errored out or crashed. I got what I wanted on the PM and picked up some tidbits on the SM as well. Given that there is still plenty of all four loans available at nearly 5pm, I really don't see the problem.
|
|
james
Posts: 2,205
Likes: 955
|
Post by james on Oct 13, 2016 18:10:28 GMT
It's purely down to FFF overload at 4pm I think. This sort of FFF activity DOES NOT SCALE (ask FC, heck ask anyone who deals in real time systems, queuing theory etc.) It can scale well enough. Just depends on having a suitable system design. What MoneyThing has is low load compared to event booking setups or what the global top thousand to top ten sites have to deal with. One often easy "cheat" is to allocate a certain amount of the available inventory to each front end cluster so they don't have to go to the ultimate source of availability. When each cluster runs out it asks for more in a chunk sufficient to fill several orders. The committing of the purchases can be queued and batched for more efficient transaction processing, if necessary. A particular buyer might get unlucky and be tied to a cluster that runs out before the rest but that's just pot luck and not an issue if it's happening with reasonable granularity. Cluster can vary in definition, it could be a set of database servers doing middle transactions before final committing to the central store or it could just be web front ends. Since money is fungible it's a much easier problem than some aspects of event booking. Though something simpler might be doable as well, depends on just where the bottlenecks happen to be. Tossing some SSDs at a database server can remove that as an issue, say. Or using persistent connections instead of dynamic. Whatever, depends on the details of the particular scaling threshold being hit at the time. But most of this doesn't much matter since a big part of fast fingers is lack of confidence in bid limits being sufficient to spread the buying time. Dealing with that is an easier technical and business combined fix. That's part of why I suggested two or more 24 hour (or 12 hour) bid periods to make it easier for Ed and co to go lower on the bid but still fill the loans in good time. Easiest way to solve demand peaks can be to eliminate the cause.
|
|
ali
Member of DD Central
Posts: 313
Likes: 311
|
Post by ali on Oct 13, 2016 18:23:44 GMT
It seems to me the biggest design flaw is having the PM and SM mixed together. This means that every refresh the system has to look up how much of every loan the user owns before it can reply which must take a lot of time and the vast amount of time, nobody cares. Since loans can't be sold on the SM until they have sold out on the PM anyway, there's no real reason why you couldn't have two pages one for each market. That way you could refresh the PM page much faster. It would probably be possible to host the two pages on different servers but that really would be introducing complexity and I think MT are right to avoid that unless it proves absolutely essential.
|
|
|
Post by GSV3MIaC on Oct 13, 2016 18:59:07 GMT
It's purely down to FFF overload at 4pm I think. This sort of FFF activity DOES NOT SCALE (ask FC, heck ask anyone who deals in real time systems, queuing theory etc.) It can scale well enough. Just depends on having a suitable system design. What MoneyThing has is low load compared to event booking setups or what the global top thousand to top ten sites have to deal with. <snip solution> Yes, but that's not my idea of 'scaling', that's completely redesigning the system to remove one particular bottleneck, and actually the solution you suggest (fanning out the available loan parts to multiple servers) looks a lot like my idea of 'prefunding' (fanning out the available loan parts to interested bidders)
|
|
james
Posts: 2,205
Likes: 955
|
Post by james on Oct 13, 2016 19:04:31 GMT
It can scale well enough. Just depends on having a suitable system design. What MoneyThing has is low load compared to event booking setups or what the global top thousand to top ten sites have to deal with. Yes, but that's not my idea of 'scaling', that's completely redesigning the system to remove one particular bottleneck, and actually the solution you suggest (fanning out the available loan parts to multiple servers) looks a lot like my idea of 'prefunding' (fanning out the available loan parts to interested bidders) That's how places solve scalability problems: iterate the design as required to deal with whatever the issues are as they grow. Been there, done that through to global top hundred with a design that then worked into the top ten a few months later and since then with many places needing help on the part of the stack that I mostly work with. Prefunding has some issues of its own but one of them is simply business differentiation. What MoneyThing is doing can work, just takes a suitable design. The current one needs business-side and technical tweaks to achieve their desired funding timescale while also dealing with demand. At least IMO, up to Ed and co to decide how to tweak their particular model.
|
|
|
Post by GSV3MIaC on Oct 13, 2016 19:11:16 GMT
Maybe savingstream would like to hire you? Although their problem is that the way the business is designed, they achieve 'global top 1000' or something status for 5 minutes every 3 days .. changing the business model to spread the load makes more sense (to me) than changing the system to try to handle the business model - it's not like 'all bids will be placed between 4pm and 4:10 pm' is a critical business requirement, or even a major selling point (IMO, again).
|
|