Diana the Valkyrie's Newsletter - October 2002

A hard man is good to beat

September

Earthquake! On September 23, we had an earthquake. I felt the first bump, then the second wave arrived and I watched things shaking in the room. It was the biggest earthquake for ten years, Richter 4.8.

This isn't the first earthquake I've felt, but it's definitely the biggest.

New and updated Galleries

Galleries added this month.

The Library

Stories added this month.

I put up the several parts of my new story, about Mwynwen.

Listen with Diana the Valkyrie

Nothing new

The Movie Theatre

Movies added this month.

Shopping Mall

StoneColdMail

The email service is free to members. And there is no advertising

Servers

September 5, I swapped my firewall for another one. The one I swapped out has hung twice, and given another problem once, and that's too much for my liking. I've put in another firewall (same software, just I replaced all the hardware) and let's see how that goes. As of end-September, it was running fine, no problems.

The big change to Newsthumbs this month, was making the Yenc decoding work, and also multi-part uuencoding. The result is that a lot of the large-sized pictures that I wasn't able to handle, now work just fine.

Looking back over the last 12 months

It is now the first anniversary of my move from Maryland/NYC, USA to Watford, UK. The move has been a great success, and I think that as a result of it, I'm giving better and more reliable service. Also, it's reduced my costs a little, and added enormously to convenience (shipping computers to the USA is made very difficult by the US customs officials, plus there's all the usual hazards of shipping things). Most of the servers that store stuff that people will want to access are now twinned (why not all? Some things are difficult to twin), so that I don't tend to get long outages. And there is currently only one server giving unreliability problems (Wendy, the backup Newsthumbs server, and I'm planning to swap her out soon).

One downside of the move, is that one of the rooms at my home is crammed full of computers, and the noise and heat in there is terrible. I might have to install air conditioning.

Whereas a couple of years ago, I was building 640 gb boxes in 5U cases, now I'm putting 1920 gb in 2U cases. This is partly because drive sizes have doubled, but more because I scouted around to find components that I could cram into a smaller space. I try to stay up with the latest technology, because it tends to be better and cheaper. Curiously, I haven't got any Pentium 4 machines yet, that's because the P4 can't be used in a dual-processor motherboard. Most of the cases I use now are rack-mount, whereas a couple of years ago, I was mostly using ordinary tower-type cases. I've become a convert to the convenience of rack-mounting.

A year ago, I was doing backups onto tape, in the conventional way. But the amount of data to store became so much, I had to upgrade my tape system. And when I looked into it, I found that a backup system onto hard disks would be faster (a few hours per month instead of a few days each month), easier (no cartridges to change) and cheaper. So now my main backup system is to three groups of hard disks.

The total storage on all the servers, is about 17 terabytes (17,000 gigabytes). That's pretty amazing to me. I remember two years ago, March 2000, we installed Vanessa. Vanessa filled an entire six foot tall, 19 inch cabinet, and amounted to just one terabyte of newsthumbs storage, and it was very exciting for me then to install such a big system. Now, just 2 1/2 years later, I can get a terabyte into a pizza box that is 1 3/4 inches high.

And what about the future? Maxtor have announced 320 gb drives, which will let me put 4 terabytes in my 2U cases, and I'm expecting that Linux will be able to handle such a large filesystem. If it can't, then I'll put 2 terabytes in a 1U case. Memory has become even cheaper, so I can afford to put in 2gb for helping the server give faster access. Processors keep getting faster, so I'm hoping to be able to keep up with the ever-expanding amount of newsthumbs that floods in each day

Cameras

BA has a Mavica now, also Ev.

The Newsgroups

Spams of the Month

I don't make these up. These are actual spams sent to me, which just strike me as funny. I don't include their contact details - go find your own spammers!

The Oxygenics Showerhead will enrich your shower with pure oxygen. Start your
day feeling refreshed and energized

Some of the world's most elegant hotels and resort spas, including the
Renaissance Wailea in Maui, have pleased guests for years with this
ergonomically designed showerhead.  Using the same principles as a jet engine,
this showerhead continually sucks in fresh air and infuses it into the water
creating a powerful spray of revitalizing, oxygen enriched water. The moment
you turn it on you realize this is like no other showerhead you^̉ve ever
experienced. It increases the water's oxygen content up to ten times, helping
to purify your shower water, and leaving your skin and hair cleaner than ever
before

... just like every other shower head does.

It's Not A Loan, It's A Solution

It's a loan.

Sponsorships

We currently have several running; Nicole Bass, Andrulla Blanchette, Sheila Burgess, Christine Envall, Marilyn Perret, Julia Santana, Peggy Schoolcraft, Larisa Hakobyan.

We're also sponsoring individual events, and funding athletes to go to events with grant dollars.

We're also doing free hosting and free bandwidth for many of our sponsored women. Bandwidth can mount up to a large bill when you're running a popular web site.

And we're sponsoring Heather Foster, Kara Bohigian, Priscilla Ribic, KerryAnn Allen, Linda Cusmano and Jodi Miller.

The latest Valkyrie Playoffs has just been launched. This was run in Sydney. And the winner was Michelle Gilham, with Tanya Leggatt, Anna Bellas and Monique Jordan as runners up.

Understanding the Internet - the Joy of Hex

A slightly unusual "Understanding the Internet" this month - it's about programming. I know that most people don't do programming (including some that think they do), and I suspect that a lot of people think that programming is beyond them. Actually, all it is, is explaining very very carefully to the computer, what you want it to do.

As you might guess, in order to run this web site, quite a lot of programming is needed. So I do it; it's easier to do it myself, than to explain to a programmer what I want them to explain to a computer, and then yell at them when they explained it wrong. And one of the things that happens occasionally when you're doing this, is a feeling of real pleasure at doing something a bit tricky. And this is especially true if you can find a neat, elegant way to do it.

The Newsthumbs takes quite a lot of programming. There's three main stages. The first stage, sucks in the news articles and separates any attachments. The second stage works out how spammy the articles look, and makes the main menus. The third stage makes the contact sheets (thumbnails); that's how most people access them.

The problem I had, is when people split a posting into multiple parts. They do this because (rightly or wrongly) they believe that smaller postings are more likely to get through Usenet, than large ones. I think they're right; I have a limit on the size of things my News servers will accept. So for example, a picture might be divided into three parts for posting.

If the encoding system is "Yenc", that's fine, because part of the Yenc specification tells you which part of the file each post is. But when people post using uuencode (the older, and commoner, standard) then you don't get that. Instead, people usually put that information in the subject of the posting, like this:

Subject: Really nice picture - taken 14/7, cat16.jpg (1/3)

The (1/3) tells you that it's part one of three parts. The name of the file, is given in the first part (on a line that starts "Begin"), but not repeated in the others.

The problem I had is, even though the poster presumably posted the parts in the right order (1, 2, 3), they don't always arrive on my server in that order. I might get part 2 before I get part 1.

My first take on this, was to hell with it. It's too complicated. If they come in the right order, I'll handle it; if they don't, I won't. But then I checked, and I found that they arrive in the wrong order quite frequently. If you think about it, if there's two parts posted at pretty much the same time, then by the time they've worked their way to me via a dozen other servers, they're as likely to be the wrong way round as the right way. And if there's three parts, there's only a 17% chance that they will arrive in the correct order.

So, here's what I did.

I can recognise a part 1, because it has a "Begin" and a filename to call the file by. I can recognise part 2 and subsequent, because there's a bunch of lines that start with M and are 61 characters long, and don't have any lower case - when I see that, I know it's another part (not part 1) of a uuencoded file. And I know, in both cases, how to convert the uuencoded stuff to a file. The problem is in working out how to bring the pieces together to recontruct the original file.

First of all, I de-scurf the subject line, taking out all the spaces and miscellaneous, non-alphanumeric characters. Then if there's a "(2/3)" there, or something similar, I remember that for later on.

If I see a subsequent part (such as (2/3), then I make a filename from the subject line, and stick "02" on the end of the filename. So now it looks like this:

Reallynicepicturetaken147cat16.jpg02

And I make a list, under a heading of "Reallynicepicturetaken147cat16.jpg", that includes that file name. This is because I haven't seen part 1 of this file yet, and I'm going to have to go back and rename them when part 1 does turn up.

When I do see part 1 of the file, then I know it's the same file because the de-scurfed subject line matches. And now I know what the filename ought to be, so I can rename the part 2 file. The filename might tell me that the reconstituted file is supposed to be called "Red Locomotive.JPG". And if I get a part 3 or subsequent parts, then I already know what I need to name that file as. Except that I de-scurf the filename, and it becomes "redlocomotive.jpg".

So I wind up with files named redlocomotive.jpg, redlocomotive.jpg02, redlocomotive.jpg03 and it's easy for me to reassemble them back to the original file.

The key trick in doing all this, is using the de-scurfed subject line as the way to tie the parts together, and in building up a list under that as a heading before I know what the final filename is supposed to be.

This is only a small part of the total Newsthumbs system of programs, but it's a part I only just added, and which is neat, elegant and therefore gave me pleasure. The Joy of Hex.

The Clubhouse

New Message Boards:

• Storm
• Shanice

In the Chatroom

Chatter of the month

Shanice heads the list this month, by a wide margin. Then comes Crystal, followed by the pack.

On the Message Boards

In September, we had 4243 posts to the boards.

Back Page

I checked the site statistics that Sandra counts up each night.

At the end of September 2002, there were about 584,000 pictures (31 gigabytes), 80 gigabytes of video, 6200 text files (mostly stories) and a total of about 112 gigabytes.

Newsthumbs is now 120 million files. More than 3 terabytes of pictures. But I'm installing 1.9 terabyte servers now :-) So far, I have four of those monster servers.

There's now six newsthumbs servers.

server	Million pictures	gigabytes	Million Text files	gigabytes	Million total files	gigabytes
Older 1	7.7	471	11.6	29	19.6	545
Older 2	7.5	497	11.2	25	18.7	522
Older 3	12.0	839	19.0	43	31.0	884
Older 4	8.0	573	13.8	29	21.7	603
Older 5	8.5	687	13.7	34	22.2	722
Latest	2.6	290	3.8	6	6.4	296
Total	46.3	3357	73.1	166	119.6	3572

To the Magic Carpet