|
|
Post by chris on Sept 25, 2014 12:49:26 GMT
Some of you may have seen the news articles today mentioning Shell Shock as being the new Heartbleed vulnerability. Our servers all run on Linux and use bash so are vulnerable at a low level but we do not think there were any routes to exploit that vulnerability due to the particular software stack we use which shouldn't have exposed any mechanisms via which we could have been attacked.
That said we have now patched all of our servers and are rebooting each in order to make sure that the changes take full effect. As this message is being posted we are rebooting the database server so there will be a window of a minute or two during which the website will be unavailable. Sorry for the lack of prior notification of the reboot, I felt it more important to make sure the vulnerability were nullified without advertising what we were doing.
Thank you for your patience.
|
|
|
|
Post by chris on Sept 25, 2014 12:57:47 GMT
|
|
|
|
Post by Ton ⓉⓞⓃ on Sept 25, 2014 13:00:08 GMT
Some of you may have seen the news articles today mentioning Shell Shock as being the new Heartbleed vulnerability. Our servers all run on Linux and use bash so are vulnerable at a low level but we do not think there were any routes to exploit that vulnerability due to the particular software stack we use which shouldn't have exposed any mechanisms via which we could have been attacked. That said we have now patched all of our servers and are rebooting each in order to make sure that the changes take full effect. As this message is being posted we are rebooting the database server so there will be a window of a minute or two during which the website will be unavailable. Sorry for the lack of prior notification of the reboot, I felt it more important to make sure the vulnerability were nullified without advertising what we were doing. Thank you for your patience. To be absolutely safe should Lenders change PW's etc? IN EDIT 10 minutes Later: The BBC Article says, So I think I will change it.
|
|
|
|
Post by chris on Sept 25, 2014 13:00:58 GMT
Some of you may have seen the news articles today mentioning Shell Shock as being the new Heartbleed vulnerability. Our servers all run on Linux and use bash so are vulnerable at a low level but we do not think there were any routes to exploit that vulnerability due to the particular software stack we use which shouldn't have exposed any mechanisms via which we could have been attacked. That said we have now patched all of our servers and are rebooting each in order to make sure that the changes take full effect. As this message is being posted we are rebooting the database server so there will be a window of a minute or two during which the website will be unavailable. Sorry for the lack of prior notification of the reboot, I felt it more important to make sure the vulnerability were nullified without advertising what we were doing. Thank you for your patience. To be absolutely safe should Lenders change PW's etc? No need as far as I'm aware.
|
|
|
|
Post by chris on Sept 25, 2014 13:15:25 GMT
To be absolutely safe should Lenders change PW's etc? No need as far as I'm aware. Actually, let me caveat that. No need as far as I'm aware as long as you aren't using the same password elsewhere. If you share the password with another site or service then you should probably change your password just in case.
|
|
|
|
Post by jevans4949 on Sept 26, 2014 3:23:10 GMT
Interesting; the first *nix bug to hit the mass media. And so simple that even I could (almost) understand it. Surprised the NSA hasn't been using it all this time - or have they?
|
|
Mike
Member of DD Central
Posts: 651
Likes: 446
|
Post by Mike on Sept 26, 2014 4:05:56 GMT
Interesting; the first *nix bug to hit the mass media. And so simple that even I could (almost) understand it. Surprised the NSA hasn't been using it all this time - or have they? I think if one were to have been exploiting this bug for long, it would have been observed quite swiftly. My belief is that SELinux should restrict the ability for the commonly published CGI method, and even without SELinux as I understand it the bug can only allow an attacker to read world-readable (or those owned by the apache user/group) files? Similarly in the case of writing files, too. Taking over an operating system and being able to gather confidential data & edit things (I believe) would require an insecure set-up in the first place, or a user executing infected code? This is my fundamental understanding of why Linux > Windows - the permissions system is what makes it so secure... But I could be wrong in this case - please correct me. Although I too am amazed that such a glaring bug has been uncovered in such a widely-used piece of code.
|
|
|
|
Post by chris on Sept 26, 2014 5:37:50 GMT
I read somewhere that the bug was actually over 25 years old, predating the internet which is quite astonishing for such a widely used piece of code.
You are right that this bug allowed attackers to effectively have shell access with the permissions available to the web server, but once you have shell access you can try other attack vectors to escalate your permissions. Linux is definitely far better than Windows but is nowhere near perfect.
We're not running Apache nor CGI, instead using Nginx and FastCGI which as far as my understanding goes is not vulnerable to the attack. There is speculation that other services like OpenSSH or DHCP could be vulnerable although again in our case I think these are either limited to no risk. All servers were patched and tested for the vulnerability at the time of my first post on this more as a precaution than because we thought there was a serious risk.
Worth noting that I'd expect the majority of other sites that you use to be affected, not just P2P, everything. If you do share passwords please change them - This time I promise not to frown too hard at you for the practice but will be keeping an eye on you.
|
|
|
|
Post by westcountryfunder on Sept 26, 2014 12:22:02 GMT
When I came across your first post on this topic sometime yesterday afternoon I of course checked up the position on my two Linux Mint boxes. In true Linux community fashion I was pleased to find that the Ubuntu depository was well up to date and Bash had received an update, presumably a patch, earlier that morning without me even noticing.
Later yesterday I read a sugggestion that patches applied may not be fully effective. However first thing this morning there has been a further Bash update. The existence of this weakness and it not being spotted for so many years has all come as an unwelcome surprise but on the face of it the speed at which this is being tackled is impressive. But thank you for drawing attention to this and in a non-sensational manner!
|
|
|
|
Post by chris on Sept 27, 2014 6:00:18 GMT
There will be a further short period of downtime at some point between Monday 29th September and Tuesday 30th as Rackspace make some changes to their environment to make sure that everything is fully patched at an infrastructure level. Whilst this should be short and uneventful I do not as yet have any more details nor can I give a more specific time.
|
|
|
|
Post by jevans4949 on Sept 27, 2014 17:28:47 GMT
Interesting comments at the end of this article suggesting it's not a bug, more of a misfeature. link |
|
james
Posts: 2,205
Likes: 955
|
Post by james on Sept 27, 2014 18:55:18 GMT
we are rebooting the database server "the" singular? I'd expect you to patch one and fail over to it from the other, then patch the first, with no visible outage time at all. of course this implies having at least two, though I wouldn't be satisfied with only two for any business with a substantial uptime requirement. Thanks for being transparent enough to mention this! I think if one were to have been exploiting this bug for long, it would have been observed quite swiftly. My belief is that SELinux should restrict the ability for the commonly published CGI method, and even without SELinux as I understand it the bug can only allow an attacker to read world-readable (or those owned by the apache user/group) files? Similarly in the case of writing files, too. Taking over an operating system and being able to gather confidential data & edit things (I believe) would require an insecure set-up in the first place, or a user executing infected code? This is my fundamental understanding of why Linux > Windows - the permissions system is what makes it so secure... But I could be wrong in this case - please correct me. You're pretty thoroughly wrong on Windows. Windows ships as standard with fine-grained access controls. A web server or database can have access constrained to just the specific files it needs to access without the need for any add-ons like SElinux. Security is one area where Windows is far, far better in its core features than linux. What leads to Windows being more prominent is the global permissions assigned to default user accounts and its far larger market share for end users. That leads to boxes being vulnerable when end users go browsing around the web. Servers don't tend to be set up to have global access to everything for their service accounts and I wouldn't expect a competent administrator to do something like that. Similar for linux as far as competence goes. I wouldn't expect a competent administrator to give a web server global access to anything, nor a database server. Of course I do still see database servers set up to have global access sometimes. There are a lot of administrators of linux boxes who just don't know linux and even its base security model very well. It's pretty uncommon for me to see SElinux in use, even at professionally run places. For high security financial applications the minimum I'd want to see is different boxes for web servers and database servers with firewall between the two as well as use of the finest-grained access controls to allow them to do their jobs. Better still would be a middleware layer on a layer of boxes between the web servers and database server that constrains the requests that the web boxes can make to the database server and adds another layer that has to be compromised before access to the database servers is possible. Then individual user accounts on the database server that further constrain what can be done even with direct access, using stored procedures and such to broaden access from nothing only in limited ways.
|
|
|
|
Post by chris on Sept 27, 2014 19:34:14 GMT
we are rebooting the database server "the" singular? I'd expect you to patch one and fail over to it from the other, then patch the first, with no visible outage time at all. of course this implies having at least two, though I wouldn't be satisfied with only two for any business with a substantial uptime requirement. Thanks for being transparent enough to mention this! We have a single master setup and whilst we have a hot slave our main DB server's load hasn't been anywhere near high enough to worry about utilising it beyond having it there. Being a financial platform I'm more worried about losing even one transaction and ensuring we have complete internal consistency of our data than 100% uptime so we haven't yet added the complexities of automatic failover, multi-master, sharding, load balancing reads across multiple hot spares, etc. It's all unnecessary cost and complexity at a time when we don't need it, although the high availability aspect is certainly of near to mid term interest.
|
|
james
Posts: 2,205
Likes: 955
|
Post by james on Sept 28, 2014 2:30:18 GMT
I agree about accuracy and consistency, as well as the need to consider scale. Interesting to observe that so far you're at the point where you have the hardware but chose not to use it - I agree that's more of an indication of pragmatic cost/benefit than anything else. If you're not set up to do it and used to it then it would just have added avoidable risk.
|
|
|
|
Post by chris on Sept 29, 2014 19:31:05 GMT
Just to remind everyone that there will be a couple of outages tonight - one is already underway - as Rackspace upgrade and protect their infrastructure from the Shellshock vulnerability.
|
|
