|
Post by padney on Feb 28, 2017 17:52:40 GMT
So I built a funding circle bot for buying loans. I got fed up with autobid being so terrible! I like to invest in C,D and E loans. Videos make it easier, so here is a link... sorry its not that exciting. www.youtube.com/watch?v=nlUgMjZulkIOne question I have for anyone else doing similar... what sort of interval do you recommend grabbing information from funding circle at? I am going with 30 seconds, and I am hoping that doesn't piss them off. It hasn't so far! I have emailed them about this when I had a simpler program. It just sent me an email when there was a C,D or E risk loan available to buy. I told them I was scraping data from their website every 30 seconds and they didn't seem to care. They just said so long as I didn't compromise performance of their site, they were happy. This program is basically my way of trying to have half a chance of getting the higher risk and return D and E loans. Autobid basically fails at that. Personally I think its BS, autobid should trigger instantly when a new loan comes up, and only then the investors with direct access to funding circle should be cleared to get their greasy paws at it.
|
|
bigfoot12
Member of DD Central
Posts: 1,817
Likes: 816
|
Post by bigfoot12 on Feb 28, 2017 20:32:33 GMT
I agree that autobid could be a lot better, it is a shame it isn't. I would think that once every 30 seconds is fine. Have you been able to buy enough Ds and Es with that setting? Mine is too slow now and hardly buys any Es and sometimes misses Ds as well, and as I am reducing my FC holding it isn't worth the rewrite I was planning.
Have the layout changes in the last few days caused you any problems - there seem to be random numbers of div additions or subtractions.
I'm not sure that I would agree with your claim of robustness. I think that you should load the loan part csv file when your program runs, or something similar. Speaking from bad experience were multiple instances all ran at the same time on something similar (not P2P) each of them did the same thing. Now the first thing I do In code like this is reconcile my internal database with the CSV download.
|
|
|
Post by padney on Mar 1, 2017 8:57:41 GMT
The comment regarding robustness is fair. I am not too fussed though in this particular case. For that to happen a) the my live bids check on the bid screen would have to fail and b) I would have to be running multiple instances (or close and reopen the program whilst loans matching the criteria are still there). I could always sell them if I had to anyway! I will bear it in mind for other applications though.
I find its fast enough for C and D, it seems to get the majority of E's that the scrape notices, but sometimes the last click of the bid button is too late. (I am normally bidding in the last 70-90% of an E loan by the time I get to it). I bid three times generally, with a 1 second interval between bids. It shows how tight everything is when the maximum time my code can take to finish bidding after a loan appears at the worst possible time is ~45 seconds, and even that isn't always quite fast enough. Simple solution is to check for new loans every 15 seconds, but I can see that starting to annoy FC.
Website changes are an annoyance, but pretty quick to fix. I just start up the program in visual studio and look at the debugging outputs I have put throughout the website interaction bits. Generally its just a case of finding out where it fails, then putting in an updated xpath. I have got that down to a pretty quick process.
|
|
|
Post by padney on Mar 1, 2017 8:58:23 GMT
Out of interest, does your code function in a similar way to mine? Is there any obvious facility you think I might be missing?
|
|
bigfoot12
Member of DD Central
Posts: 1,817
Likes: 816
|
Post by bigfoot12 on Mar 1, 2017 11:18:55 GMT
I mainly use xpath so in that sense it does.
Mine started out very similar to yours as an improved autobid; it would buy a bit of everything except 6% A+. I also saved as much data as possible to see if I could better select which loans to invest in, but this hasn't been as useful as I hoped. Then over the last 18 or so month I added extra features for things like cashback, which aren't used much any more. But I still use some of the functions in a semi automatic way for putting loans up for sale.
Mine is very slow and rarely buys an E these days, and I am scaling back on FC so it isn't worth the time to fix it. My FC holding is less than 1/3 of what it was a year ago.
|
|
|
Post by padney on Mar 1, 2017 18:11:21 GMT
Fair enough. What are you going into now?
I have done some analysis on trying to look at historical default rates on the loanbook by industry, loan purpose ect.. and have not come up with anything particularly useful in that regard.
|
|
michaelc
Member of DD Central
Say No To T.D.S.
Posts: 5,707
Likes: 2,983
Member is Online
|
Post by michaelc on Apr 26, 2017 18:28:45 GMT
Nice! What is it written in?
I have written one in python without any UI. I've used selenium but am not altogether happy with the performance. Initially I found some of the E loan parts were being bought with much higher frequency than me. I've since upgraded my software to use multiple child bots that trigger when a loan becomes available. I haven't really had the time to assess performance yet though but I wonder if those bots that can snap up dozens of E loan parts are analyzing the http protocol and resending that way so that they don't have to go through slow browser code (headless or otherwise). One problem I do have is determining whether a purchase has been successful. I started by reading the balance but I don't think it gets updated quick enough so I just try to buy a maximum amount and see what I'm left with after the frenzy. Doping it via http directly or using the browser to send an xhr ajax request might overcome that.
|
|
|
Post by spiker on Apr 27, 2017 19:14:24 GMT
Just wait for the popup element confirming the bid?
wait.Until(ExpectedConditions.ElementExists(By.XPath("//li[@class='notification']")));
|
|
|
Post by yorkshireman on Apr 28, 2017 14:11:13 GMT
Mods: Could we have this thread translated from Serbo-Croat to English please?
|
|
maxmarengo
Member of DD Central
Posts: 96
Likes: 28
|
Post by maxmarengo on Apr 30, 2017 12:47:25 GMT
Nice! What is it written in? I have written one in python without any UI. I've used selenium but am not altogether happy with the performance. Initially I found some of the E loan parts were being bought with much higher frequency than me. I've since upgraded my software to use multiple child bots that trigger when a loan becomes available. I haven't really had the time to assess performance yet though but I wonder if those bots that can snap up dozens of E loan parts are analyzing the http protocol and resending that way so that they don't have to go through slow browser code (headless or otherwise). One problem I do have is determining whether a purchase has been successful. I started by reading the balance but I don't think it gets updated quick enough so I just try to buy a maximum amount and see what I'm left with after the frenzy. Doping it via http directly or using the browser to send an xhr ajax request might overcome that. I use Selenium - great for testing because you can see what is going on - and can understand your performance concerns. What Python libraries do you use instead of Selenium?
|
|
|
Post by grahamreeds on May 4, 2017 11:13:30 GMT
I would be interested in this too as I was thinking of writing a bot just to learn Python. I want to automate parts of my daily FC routine.
|
|
|
Post by spiker on May 4, 2017 12:14:05 GMT
Its a lot harder of a task if you don't use Selenium because then you have to ensure you structure your HTTP requests correctly (form data, cookies, headers etc). Logging in etc is easily achievable, but purchasing the loans with direct HTTP requests gets a little messy... i.e. there's a lot of fact finding involved to observe how the HTTP requests are structured. Which isn't ideal as this isn't a test site, so your fact finding involve actually buying loan parts... (Though I suppose you could buy one part manually and record the HTTP traffic)
If I was you i'd stick to perl (or alternative language) combined with selenium...
|
|
bigfoot12
Member of DD Central
Posts: 1,817
Likes: 816
|
Post by bigfoot12 on May 4, 2017 12:27:26 GMT
I think that you can use Python with Selenium if you want to. I haven't tried, as I don't know Python.
|
|
maxmarengo
Member of DD Central
Posts: 96
Likes: 28
|
Post by maxmarengo on May 5, 2017 15:11:14 GMT
Three things I like about Selenium: - You can program in a range of languages (including Python and Java) so you can make use of skills you already have.
- You can run it through a browser you are familiar with (Firefox, Chrome) , see the steps your program drives and use the browser debug tools.
- It is free.
There may be better, faster, more high powered tools that other Bot writers are using but for someone who is doing it to learn some new skills, it is a good place to start!
|
|
stub8535
Member of DD Central
personal opinions only. Not qualified to advise on investment products.
Posts: 1,447
Likes: 945
|
Post by stub8535 on May 5, 2017 15:20:53 GMT
You also have the disadvantage, on one particular site, that you are the wrong side of the user interface. But hey, sites refuse to accept there is an issue at times. Only way is to play the same game or leave the field for the next schmuck to get scalped until sites do something about it. The profit margins made by using these methods are eyewatering and free from tax plus legal.
|
|