The Magnet Blog

nirav April 30th, 2006

I wrote about comment spam I was getting on my blogs a while ago. I didn’t bother to do anything about it, but last week it just went out of hands.

I used to get about 250 spam comments everyday on both mehtanirav and TrustAndFreedom. The Magnet blog also gets lot of spam.

I didn’t want to install a Captcha on the blog as the server does not have GD support. And I fould WP-HashCash. It’s pretty simple to setup, does not require GD, and is 100% effective! I did not have any spam for the last three days!

I strongly recommend HashCash for anybody using WordPress!

• Blogs
• Comments(0)

nirav April 29th, 2006

When came to office on Saturday, I wanted to spend some time on OpenCerti and complete the pending ActionScript work. But there was a problem waiting! The emails were stuck, and we were getting duplicate emails.

Let me give you some background. We have a set of Linux servers in the office. The magnet-i.com site is hosted in a NOC in USA. We have a few POP accounts on the server and a catchall. The catchall captures emails for most of the users. POP accounts are primarily used by people who travel and need roaming access etc. We download emails to local server using Fetchmail. The internet connection is via cable modem and DSL.

Now this kind of problems have happened before. I remember in the dialup days, if we had a large number of emails, and if the internet connection drops while fetchmail is running, it would start downloading all the emails again the next time you connect. Resulting in duplicate emails.

This time around, we have a major recruitment drive going on. The catchall account was more than 85MB in size. And fetchmail was giving up on it. It was not only the size, but also the number of messages that was big. And the mails were not only for the HR, but also for other people in the organization.

I and Vishal looked at the problem and first tried to delete unwanted mails via webmail. We got it down to 55MB, but this was still too big for fetchmail to handle on our internet connection.

The next step we generally take is to take the mbox file, bzip2 it, download it over web, append to the local mbox, and let the mails flood into the email client. This time, we couldn’t do it because it was not only one user account that the emails were destined too. But this is the track that we take up.

The mbox file contains all the mails in a single file. So we SSH’ed to the server, located the catchall account (something like /home/user/mail/domainname/emailaccount/ on cpanel servers) mailbox. Doing a bzip2 on it, got the file size down to 11MB. (OT: I wonder why they didn’t add compression in POP/SMTP. That would have saved a lot of traffic).

Downloading this via web (so that we can resume the download if it gets broken) did not work. Somehow our server did not allow direct file downloads. (Guess it was me only who disabled hot links like this..) We didn’t have too much time to go reconfigure the server, so we simply FTP’ed the file to one of our servers that we use for exchanging files with clients. Downloaded the 11MB file to local machine and posted it on the mail server.

Now what?

Idea! What if we configure fetchmail to connect to the local POP server? We could create a new user and push the downloaded mails into the new user’s mbox file. When fetchmail connects to the local server to fetch emails, it will fetch emails from this new account. And it can deliver them to the local users as per the original configuration!

The idea was right, and we tested it with one or two messages in the mbox file. First it bounced back, saying there’s a “mail forwarding loop”. We removed the “no dns, aka magnet-i.com” part from the fetchmail config, and it started pushing the emails to the postmaster. At least it did not bounce back! A few trials later - and inspecting the logs and the postmaster emails - we figured it out. We need to have the “aka magnet-i.com” line in, and have the mbox file of the server.

So set this up, ran fetchmail, and it went chopping the mbox like crazy and delivering emails to the local users. If we used some other method (like downloading only new messages with UIDL etc), it would have taken 5 times more time to download the emails, and we would have to monitor the process.

This gets us the best practices for handling large emails that are stuck on the server.

• Bzip the mbox file. Download it via web.
• Unzip the mbox file on local server. And process it there.
• If it’s a single email account, simply append the mbox file to local mbox file. And let the user dowload emails via her email client.
• If there are multiple email accounts in the mbox file (mbox of a catchall), create a new account on local server and append the mbox to it. Add a rule in .fetchmailrc to use local server as POP3 and fetch emails for the new account created. And then distribute it to “* here”
• You can use “fetchmail -v” for verbose output.
• Monitor /var/log/maillog and /var/log/fetchmail.log for info. Also check the postmaster account (we run Postfix) for error reports.

Good troubleshooting for the day! I am off to something bigger now!

• GNU/Linux
• Comments(0)

nirav April 29th, 2006

For doing file uploads with Flash, you have to use the FileReference class. Now it’s a really nice feature - provides the basic features for uploading files and tracking their progress. There are two things I want to note about it on this post.

1. When the file is uploaded from Mac, the “type” property is not available.
The FileReference class has a property “type”, that gives you the extension (with the dot) of the file. This is very handy to check file types and reject the ones you don’t want to allow. But when you upload a file from the Mac, the “type” property is blank. I am not sure if this is a Mac issue or a Flash issue. But as of now, the best way for you to find the extension of the file is to substr it.

var fName = f.name; // f is the FileReference Object
var ext:String = fName.substr(fName.lastIndexOf(”.”), fName.length);
ext = ext.toLowerCase();

2. You can’t return data from the server to Flash when upload completes
When you provide file upload feature in your application, many of the times, you want to return an ID of the uploaded file or its URL. The FileReference class does not support this. You have to make another call to the server to get any additional information. The FileReference will simply give you “onComplete” event when the server returns HTTP status code 200. It discards any data you send from the server in the response.

It would be great if we can pass a url encoded string back and the values were available to the onComplete event.

• Flex & Flash
• Comments(0)

nirav April 27th, 2006

Hey, there is a BarCamp in Mumbai! BarCamp is an unusual conference! It’s ad-hoc, people gather and share about technologies they like. People listen to and speak.

It’s going to be the first BarCamp I will visit, but I have heard lot of good things about BarCamps. So it’s going to be fun. I am planning to speak on our Flex and OpenLaszlo experiences there!

Are you in Mumbai? Then be sure to check out BarCamp Mumbai, and participate!

• General , RIA
• Comments(0)

nirav April 27th, 2006

After Chitralekha did a story on us (GujaratiLexicon and Utkarsh, Chitralekha site requires special font download to view the site in Gujarati); we have been getting lot of inquiries. Comments have ranged from thrilled Gujarati’s, to upset people who couldn’t get the free CDs to people who put the CD in a VCD player.

Then there is another set of people. People who’ve been working on the language for years and are happy to see this happening. Ratilal Chandaria was one such person, who came down to meet us specially six months ago. That meeting started a new journey for Utkarsh. We shook hands with GujaratiLexicon and now taking full care of it.

This time too, we are meeting many such people. Upendra Dave from London was impressed and wants to do something in the areas of learning Gujarati with electronic media. Mayank Thakkar came from Mahuva and would like us to do something in Education. And then there are many other people that either Ashok or Kartik met.

It’s an honor to meet people who have given their life to the language. Gives us energy to keep pushing this endeavor.

There is lot more that’s coming on Utkarsh now!

• ગુજરાતી - Gujarati , General
• Comments(1)