Diana the Valkyrie's Newsletter - February 2011
A hard man is good to beat
The weather is back to normal now; cold and drizzly.
Galleries added this month.
Stories added this month.
Movies added this month.
Running well. I've moved the current Newsthumbs to the archive, and set up a new server for the current.
I did the server move.
First, I went to the colocation site and installed 17 servers. It would have been 18, but one failed under test. I later determined that it was a failure of the CPU, which I wouldn't really have expected to happen. Usually its a CPU fan, or a power supply, or memory. Occasionally the motherboard. But CPUs go on for ever. But this one didn't.
Anyway, 17 servers is enough. I have four big customer-facing servers, with four terabytes in each box. There's also an even bigger backup server (12 terabytes), which daily copies what's on the customer-facing servers. And there's a similar backup server behind that.
That's six servers, you might wonder what the other 11 are. Well, they're backup spares. They're powered off, to save electricity (and that's not just me being green, it also saves money). But I can power them on in a minute or two if one of the hot servers fails.
On top of the stack of servers, there's the ethernet switch and the firewall. And down the side, there's the remote power switches; that's what lets me have the spare servers powered off until I need them.
So I went down there with a carload of servers, and they helped me put them in the rack, made the cables nice and neat (left to myself I'd make a mare's nest). They all powered up when I tested them, but when I got home, one of them (the backup of the backup) was missing a drive, so I had to go back and fix it. It turned out to be a loose card, and bit of cardboard wedging it was the answer.
First, I configured the firewall. I wanted to do that while I was actually there, because if you make a bad mistake while you're doing that, you can suddenly lose all access to *everything* because you just told the firewall to shut you out. But I didn't blunder like that. I also made sure that the remote power switches were working.
Interestingly, minicom, which is the serial program for linux, was able to talk to the firewall, but couldn't talk to the remote power switches. I have no idea why.
But the physical servers, are only part of the story. There's also all the data. And, as you can imagine, there's a lot of it. Several terabytes. I'd prepared for this earlier, by loading the servers up with all the files while they were here with me. But since that load-up, there had been updates, and I had to get those also.
I started off by moving eight sites. I disabled ftp to those sites, so they couldn't be updated while I was moving them (if they had, then the updates wouldn't have been on the new servers). That went so well, I was very pleased; I'd been rather dreading this part, even though it's the easiest part of the data move. And I changed the DNS on the old DNS server so that people would be accessing the new site, and then I watched the logs as accesses to the old server gradually died out, and access to the new server replaced them.
So then I moved the extremely large Herbiceps site (and related). That also went well. So next I moved the large assortment of small sites (small meaning less than 100 gigabytes). And that worked too. So, with all that looking good, I moved the Diana the Valkyrie site.
There were a few problems with all this; I discovered that there's a nifty thing that you have to do to make ssh happen quickly (I'm using scp for moving files around). I discovered that I hadn't configured Apache quite right; I needed to disable scriptalias and to enable server side includes. I found that I'd left off the "remove one person's account" facility, and other little stuff like that. But all in all, no big problems.
Then I needed to change the DNS. I was dreading this part - I really didn't know exactly how to do it. I went on to the godaddy.com web site and tried to tell it that all the nameservers were different now, but it wouldn't accept the new nameserver - it said it wasn't registered. After an hour of fuffing about, I decided to go to the Network Solutions web site, which is where the main sites are registered, and do the changes there. That worked, and then when I went back to godaddy, it accepted the new nameservers. I think I understand what happened ... I think. It's because I have some servers with one registrar, and some with another.
And I also had to do a bit of messing around to reconfigure email, and I set up a thing that lets me look at my bandwidth usage, and I can also see my electricity usage, which is 3.2 amps, and that's not bad, I'd estimated 4. I don't know the bandwidth usage yet though, because I'm still doing huge amounts of file copying to get the backups straight, and that makes it look big now, but it won't be like that in the longer run. But it's looking good, and I can always get more bandwidth if I need to.
Then I tackled a biggie - Usenet. That's the basis of the Newsthumbs, and it's huge. I get a feed from a service provider, and you can't phone them. Well, you can ... but they never call you back. It all has to be done via email, which is a bit hairy for something this complex.
The first email I sent them was fielded by front line tech support, who thought I was one of their ordinary web-access users, and he said that changing my IP address wouldn't be a problem. But I persuaded him that I wasn't one of his $10/month customers, that I was having a BIG feed called an "IHAVE", and he realised that this needed putting over to someone more technical. That went well, and the second person I was emailing, understood the issue, and told me that when I was ready, they'd make the change.
In my experience, anything to do with Usenet, isn't simple. For example, my news server, which is Dnews, had just started announcing that the licence had expired (which it hadn't), and I had to get a new licence key from New Zealand. But that went very easily. Whew!
And to my surprise, the switch of news server went easily too. I emailed them with the new IP address, and after a few minutes, my new news server started telling me that it was getting files. So that seems to be working OK. One minor problem so far - I'd forgotten to create a directory to store the checklists and zip files for download, so that feature wasn't working, but someone told me about the problem, and it was fixed in a trice.
So, with the news server working, it was time to start up the news digester; the thing that creates the newsthumbs system. For my first try, I used a spare server. That worked well, but (as expected), it used a *lot* of CPU, that's why I didn't really want to run this on a customer facing server. Because it would have made that server look slow.
Then I had a great idea. I'm running two backup servers, to make sure there isn't data loss. But those aren't customer-facing, and although they need oodles of disk space, their CPUs don't get much use. You can see the idea - use those as the news digesters. So I tried that, and that worked well. Which means I can now shut down the spare server I'd started up before, which saves electricity.
And now, I think, it's all complete. Eleven days from now, on January 26, I have to shut down the servers at my old colocation and bring them back here, for possible reuse in future. But that was be easy - it was just a matter of unplugging everything and piling them into the car.
But ... they had a final hassle for me at Watford. I'd arranged the visit, and had an email confirming that it was all OK. But the security people at Watford hadn't been told, or hadn't been told properly, or something. Showing them my email didn't help (I could have forged that easily). So I suggested they phone the people who set up these visits in their company. I was astounded to learn that they weren't allowed to do that.
Eventually, it was sorted out, and I removed all the equipment from there.
The big test is, did you notice that all this was happening? I would hope not!
And then on January 18, I had a power cut at the Valkyrie Data Shed. This went on for a couple of hours, and although all my UPSes worked, a couple of hours is too long for them. So I got the generator out and fired it up. What I discovered, is that the generator (a cheap one that's supposed to give me 3 kw) isn't very good at voltage. So although the UPSes got some help from it, they were still having to work a bit. And, one by one, they pegged out.
Bringing all the computers back up again was, of course, a bit of a nightmare. And then I discovered that two of my DSL routers had decided that enough was enough, and they weren't going to route any more. Fortunately, I had spares on-site, and was able to replace them.
A busy month!
Here's the full list of DtV family web sites
Member | Posts |
| buffy18976 | 11 |
| gaily304 | 8 |
| gee1407 | 3 |
| birkjen | 3 |
The chatroom is pretty much dead.
74 posts this month. On Sergeant Wick and PFC Kandor's Crush Camp, a fascinating look behind the scenes of a scissors web site.
Most posted Board of the month | Poster of the month | ||||||||||||||||||||||
|
|||||||||||||||||||||||
| Buffy's Pub died |
Very little activity |
Mavis is counting the number of times the message list is checked for each board. This gives a very different picture from the one above.
Most listed Board of the month | Most read Board of the month | ||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||
| Buffy's Pub died | The scissors board is still active. |
I checked the site statistics that Sandra counts up each night.
At the end ofJanuary 2011, there were about 1,281,000 pictures (229 gigabytes), 417 gigabytes of video, 13400 text files (mostly stories) and a total of about 650 gigabytes. There's about 455 million pictures altogether in Newsthumbs, increasing at about 5 million per month.