X-POP3-Rcpt: jaques.law@louise Date: Fri, 11 Apr 1997 13:06:57 -0600 From: Jim Nickel Subject: DLC-West Information Update Whew! What a time we have had. Let me start by saying that we sure do appreciate the patience you have had with us while we recovered. Thank you! I will start by giving a brief run down on what has happened, and what will be happening soon. Then I will give the details of our adventure. Since this is a informational message only, those of you that don't wish to read the details may safely skip this letter. What happened in brief: - we were vandalized by an individual from Arizona - he erased our main web server and a dial-in server - this person has been caught and equipment confiscated - this was not a run-of-the-mill attack - our tape backup unit failed - about 10% of our modems suddenly stopped working - one of the erased hard drives stopped working What we have done about it: - traced the individual back with evidence so that he would be caught - we restored dial-in service by midnight that same night - performed advanced data recovery on the drives - restored full service (web sites, online features) over the next 2 weeks - added new phone lines - got replacement modems for the bad ones - purchased a totally new backup system - installed new dial-in server equipment - installed newest versions of software - beefed up security Now to the details... Back in Jan/Feb. we noticed a connection coming in from Arizona. Since we are being used by people from all over the world, we did not give it much thought at first. Then we noticed that this person was trying some unusual things. Sending bizarre cryptic commands from his web browser and attempting to telnet into our computers. We began tracking these attacks. We also put into place all the known protections that we had available. One problem was that some of them required us to take the entire system down in order to implement them, thus we were saving those for a later time (they are now obviously in place). Another problem was that while he had every night for weeks to make his attempts, we had a limited amount of time to put into keeping our system safe. We also HAVE to leave ourselves vunerable to a certain extent....if we put our main web servers and dialin servers behind a firewall, then your access would be restricted, and outside people would not be able to see our web pages. So after many attempts, our "friend" finally did get access around March 15th. At this point we were only interested in getting this person to stop. We could not fathom what his motivation would be to spend so much time getting into our system (unless he was hired by our competition?). We then contacted the provider where he was getting his access. The SYSOP (system operator) in Arizona said that he had had complaints before and he suspected he knew who it was. He asked for some proof of these attacks. We provided him with some. He then gave the user he suspected a "static IP" address. Each computer on the Internet has a different IP address so that the Internet knows where the data should go. Normally you have a "dynamic IP" address...that is it is different every time you log on. That is because we don't assign an IP address to a person but to each phone line, so depending on which line you connect to you get a different IP address each time you call. However, we do have the ability to assign a static IP for a particular user. By assigning this user a static IP address, there is not as much annonimity and so we have evidence of what was done. Now we tracked his every move on our system. We moved copies of the evidence off to other machines. We also got his phone number and attempted to call him. It was either busy or only a answering machine. I was unwilling to simply leave a message. As you might imagine, this caused us sleepless nights. We then contacted the Arizona District Attorney's office. They asked for copies of the evidence which we provided. They told us that this was a very tight case and they would get a search warrant right away. Unfortunately, our vandal must have discovered that we were tracking him and decided that the only way to cover his tracks was to erase all the hard drives he could get his hands on. If we had been able to get him to stop, it would probably have ended there, but with this last act, he sealed his fate. We now spent the next day putting our system back together. We managed to restore dial-in service by midnight that same day. This was after finding out that our 8mm tape backup was not backing up quite so well as we had thought. We then spent the next week attempting to restore data from tapes...all to no avail. We enlisted the aid of a data recovery specialist and he was able to retrieve some things. Ultimately, it was better to have customers re-upload their pages then to spend any more time on trying to recover them. (Thanks everyone!) Of course, when it rains, it pours. Thus this was the time when modems stopped working and hard drives crashed. We also needed more phone lines. Doing things as quickly as possible, we ordered and received replacement modems from Cardinal while ordering new ones as well. Naturally, they shipped us the wrong ones and we ended up playing modem tag for a while. Sasktel did their part to liven things up. While installing the new lines, they messed up the programming and also put the new lines into service before we had any equipment to attach to them. They claim they don't pick on us, but it sure feels like it. In any case, the District Attorney's office got the warrant and went to his house. The 21 year old fellow was woken up and talked to. His equipment was confiscated. During the time the police were at his house, he confessed to the crime on tape. They are now going through all his files to see if there are other victims. Since then, we have purchased a completely new 4mm DAT drive with Auto-Loader (it can store up to 96 GIG on twelve different tapes!), put new hard drives in, installed a completely new router, improved our administration program, installed a program to watch our phone lines and alert us to ones that are misbehaving. We have also improved service by switching to our own news server. It is updated very frequently and can be accessed by using your favorite newsreader program and pointing it to news.dlcwest.com. We also installed a new router. In the coming weeks and months we are adding more lines, adding additional bandwidth, and revamping our web pages with new features like classifieds. We welcome your suggestions on features you would like to see. We are working hard to make the system better and more reliable for you. We are committed to providing an Internet service that you can happily recommend to your friends. We thank you for "sticking with us" during the difficult time we just went through. We would not be here without you. If you have any questions about our service, please do not hesitate to call or write us. Sincerely, Jim Nickel President DLC-West Internet jim.nickel@dlcwest.com